FAQ
Hi,

I have tried to running CM4 to install everything onto 6 nodes but get
heatbeat failed messages along with other install failure messages.
Please find attached my errors and some more info.

There is no firewalling on our LAN.

all done
closing logging file descriptor
IOError: Port 9000 not free on 'cloudera-node2'

[12/Feb/2013 10:38:31 +0000] 23655 MainThread _cplogging ERROR
[12/Feb/2013:10:38:31] ENGINE Shutting down due to error in start listener:
Traceback (most recent call last):
File
"/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/process/wspbus.py",
line 235, in start
self.publish('start')
File
"/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/process/wspbus.py",
line 215, in publish
raise exc
ChannelFailures: IOError("Port 9000 not free on 'cloudera-node2'",)

[12/Feb/2013 10:38:31 +0000] 23655 MainThread _cplogging INFO
[12/Feb/2013:10:38:31] ENGINE Bus STOPPING
[12/Feb/2013 10:38:31 +0000] 23655 MainThread _cplogging INFO
[12/Feb/2013:10:38:31] ENGINE HTTP Server
cherrypy._cpwsgi_server.CPWSGIServer(('cloudera-node2', 9000)) already shut
down
[12/Feb/2013 10:38:31 +0000] 23655 MainThread _cplogging INFO
[12/Feb/2013:10:38:31] ENGINE Stopped thread '_TimeoutMonitor'.
[12/Feb/2013 10:38:31 +0000] 23655 MainThread _cplogging INFO
[12/Feb/2013:10:38:31] ENGINE Bus STOPPED
[12/Feb/2013 10:38:31 +0000] 23655 MainThread _cplogging INFO
[12/Feb/2013:10:38:31] ENGINE Bus EXITING
[12/Feb/2013 10:38:31 +0000] 23655 MainThread _cplogging INFO
[12/Feb/2013:10:38:31] ENGINE Bus EXITED
[12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO No command line
vars
[12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Agent starting
as pid 24672 user root(0) group root(0).
[12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Re-using
pre-existing directory: /var/run/cloudera-scm-agent
[12/Feb/2013 10:46:12 +0000] 24672 MainThread downloader INFO Downloader
path: /opt/cloudera/parcel-cache
[12/Feb/2013 10:46:12 +0000] 24672 MainThread parcel_cache INFO Using
/opt/cloudera/parcel-cache for parcel cache
[12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Re-using
pre-existing directory: /var/run/cloudera-scm-agent/cgroups
[12/Feb/2013 10:46:12 +0000] 24672 MainThread cgroups INFO Reusing
/var/run/cloudera-scm-agent/cgroups/memory
[12/Feb/2013 10:46:12 +0000] 24672 MainThread cgroups INFO Reusing
/var/run/cloudera-scm-agent/cgroups/cpu
[12/Feb/2013 10:46:12 +0000] 24672 MainThread cgroups INFO Reusing
/var/run/cloudera-scm-agent/cgroups/blkio
[12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Found cgroups
capabilities: {'has_memory': True, 'default_memory_limit_in_bytes':
9223372036854775807, 'writable_cgroup_dot_procs': True, 'has_cpu': True,
'default_blkio_weight': 1000, 'default_cpu_shares': 1024, 'has_blkio': True}
[12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Setting up
supervisord event monitor.
[12/Feb/2013 10:46:12 +0000] 24672 MainThread filesystem_map INFO
Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs']
[12/Feb/2013 10:46:12 +0000] 24672 MainThread filesystem_map INFO Using
timeout of 2.000000
[12/Feb/2013 10:46:12 +0000] 24672 MainThread filesystem_map INFO Using
join timeout of 0.100000
[12/Feb/2013 10:46:12 +0000] 24672 MainThread filesystem_map INFO Using
tolerance of 60.000000
[12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Using
metrics_url_timeout_seconds of 30.000000
[12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Using
task_metrics_timeout_seconds of 5.000000
[12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Using
max_collection_wait_seconds of 10.000000
[12/Feb/2013 10:46:12 +0000] 24672 MainThread metrics INFO Importing
tasktracker metric schema from file
/usr/lib64/cmf/agent/src/cmf/monitor/tasktracker/schema.json
[12/Feb/2013 10:46:12 +0000] 24672 MainThread throttling_logger ERROR
Failed to collect disk stats for device 'mapper/VolGroup-lv_root'
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/src/cmf/monitor/host/disk_device.py", line
54, in collect_update
stat_file = open(self._stat_path)
IOError: [Errno 2] No such file or directory:
'/sys/block/mapper/VolGroup-lv_root/stat'
[12/Feb/2013 10:46:12 +0000] 24672 MainThread disk_devices WARNING
Failed to add DiskDevice for disk='mapper/VolGroup-lv_root'
[12/Feb/2013 10:46:12 +0000] 24672 MainThread __init__ INFO Importing
metric schema from file /usr/lib64/cmf/agent/src/cmf/monitor/schema.json
[12/Feb/2013 10:46:14 +0000] 24672 MainThread agent INFO Supervised
processes will inherit the following environment: {'CMF_CONF_DIR':
'/etc/cloudera-scm-agent', 'CMF_AGENT_HIVE_DEFAULT_XML':
'/etc/hive/conf.dist/hive-default.xml', 'CMF_AGENT_MGMT_HOME':
'/usr/share/cmf', 'CMF_AGENT_CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs',
'PATH': '/sbin:/usr/sbin:/bin:/usr/bin', 'CMF_AGENT_CDH_FLUME_HOME':
'/usr/lib/flume-ng', 'CMF_AGENT_CDH_HBASE_HOME': '/usr/lib/hbase', 'LANG':
'en_US.UTF-8', 'TERM': 'vt100', 'CLOUDERA_ORACLE_CONNECTOR_JAR':
'/usr/share/java/oracle-connector-java.jar', 'CMF_AGENT_CDH_HUE_HOME':
'/usr/share/hue', 'SHLVL': '1', 'CMF_AGENT_TOMCAT_HOME':
'/usr/lib/bigtop-tomcat', 'CLOUDERA_POSTGRESQL_JDBC_JAR':
'/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar',
'CMF_AGENT_CDH_PIG_HOME': '/usr/lib/pig', 'CMF_AGENT_CDH_HIVE_HOME':
'/usr/lib/hive', 'CLOUDERA_MYSQL_CONNECTOR_JAR':
'/usr/share/java/mysql-connector-java.jar', 'CMF_AGENT_CDH_YARN_HOME':
'/usr/lib/hadoop-yarn', 'CMF_AGENT_CDH_HADOOP_BIN': '/usr/bin/hadoop',
'CMF_AGENT_CDH_OOZIE_HOME': '/usr/lib/oozie',
'CMF_AGENT_CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper',
'CMF_AGENT_CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CMF_AGENT_JSVC_HOME':
'/usr/libexec/bigtop-utils', 'CMF_AGENT_CDH_HDFS_HOME':
'/usr/lib/hadoop-hdfs', 'CMF_AGENT_CDH_MR1_HOME':
'/usr/lib/hadoop-0.20-mapreduce', 'CMF_AGENT_CDH_MR2_HOME':
'/usr/lib/hadoop-mapreduce', 'PWD': '/', 'CMF_AGENT_CDH_HADOOP_HOME':
'/usr/lib/hadoop', 'CMF_AGENT_CDH_IMPALA_HOME': '/usr/lib/impala'}
[12/Feb/2013 10:46:14 +0000] 24672 MainThread agent INFO Supervised
processes will add the following their environment environment:
{'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'CMF_PACKAGE_DIR':
'/usr/lib64/cmf/service', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME':
'/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CDH_YARN_HOME':
'/usr/lib/hadoop-yarn', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs',
'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE
HDFS_DFS_DIR_NOT_EMPTY', 'CLOUDERA_ORACLE_CONNECTOR_JAR':
'/usr/share/java/oracle-connector-java.jar', 'CDH_HIVE_HOME':
'/usr/lib/hive', 'JSVC_HOME': '/usr/libexec/bigtop-utils',
'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml',
'CLOUDERA_POSTGRESQL_JDBC_JAR':
'/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_HBASE_HOME':
'/usr/lib/hbase', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_ZOOKEEPER_HOME':
'/usr/lib/zookeeper', 'CDH_HUE_HOME': '/usr/share/hue',
'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar',
'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_PIG_HOME':
'/usr/lib/pig', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs',
'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'TOMCAT_HOME':
'/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'}
[12/Feb/2013 10:46:14 +0000] 24672 MainThread agent INFO Re-using
pre-existing directory: /var/run/cloudera-scm-agent/process
[12/Feb/2013 10:46:14 +0000] 24672 MainThread agent INFO Re-using
pre-existing directory: /var/run/cloudera-scm-agent/supervisor
[12/Feb/2013 10:46:14 +0000] 24672 MainThread agent INFO Re-using
pre-existing directory: /var/run/cloudera-scm-agent/supervisor/include
[12/Feb/2013 10:46:14 +0000] 24672 MainThread agent INFO Connecting to
previous supervisor: agent-8767-1360595961.
[12/Feb/2013 10:46:14 +0000] 24672 MainThread _cplogging INFO
[12/Feb/2013:10:46:14] ENGINE Bus STARTING
[12/Feb/2013 10:46:14 +0000] 24672 MainThread _cplogging INFO
[12/Feb/2013:10:46:14] ENGINE Started monitor thread '_TimeoutMonitor'.

IP Tables are OFF
SELinux is DISABLED

Netstat readout

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign
Address State PID/Program name
tcp 0 0 cloudera-node1:9000
*:* LISTEN 17258/python
tcp 0 0 *:7432
*:* LISTEN 1798/postgres
tcp 0 0 localhost:9001
*:* LISTEN 8945/python2.6
tcp 0 0 *:7180
*:* LISTEN 1826/java
tcp 0 0 *:7182
*:* LISTEN 1826/java
tcp 0 0 *:111
*:* LISTEN 1426/rpcbind
tcp 0 0 *:22
*:* LISTEN
tcp 0 0 *:54102
*:* LISTEN
tcp 0 0 localhost:36186
localhost:9001 ESTABLISHED
tcp 0 0 cloudera-node1:38815
cloudera-node3:22 ESTABLISHED
tcp 0 0 localhost:7432
localhost:58248 ESTABLISHED
tcp 0 0 cloudera-node1:7180 webvpn.domain.co.uk:16971
ESTABLISHED
tcp 0 0 localhost:58245
localhost:7432 ESTABLISHED
tcp 0 0 localhost:58247
localhost:7432 ESTABLISHED
tcp 0 0 cloudera-node1:39077
cloudera-node1:7182 ESTABLISHED
tcp 0 0 localhost:7432
localhost:58246 ESTABLISHED
tcp 0 0 cloudera-node1:22 webvpn.domain.co.uk:16062
ESTABLISHED
tcp 0 0 cloudera-node1:43521
cloudera-node6:22 ESTABLISHED
tcp 0 0 localhost:58246
localhost:7432 ESTABLISHED
tcp 0 0 localhost:58248
localhost:7432 ESTABLISHED
tcp 0 0 cloudera-node1:49106
cloudera-node5:22 ESTABLISHED
tcp 0 0 cloudera-node1:49705
cloudera-node4:22 ESTABLISHED
tcp 0 0 localhost:7432
localhost:58244 ESTABLISHED
tcp 0 0 localhost:7432
localhost:58247 ESTABLISHED
tcp 0 0 localhost:7432
localhost:58245 ESTABLISHED
tcp 0 0 localhost:9001
localhost:36186 ESTABLISHED
tcp 0 0 cloudera-node1:7182
cloudera-node1:39077 ESTABLISHED
tcp 0 0 localhost:58244
localhost:7432 ESTABLISHED
tcp 0 0 *:7432
*:* LISTEN
tcp 0 0 *:33514
*:* LISTEN
tcp 0 0 *:111
*:* LISTEN
tcp 0 0 *:22
*:* LISTEN

Any thoughts on how to resolve this?

Search Discussions

  • Jean-Marc Spaggiari at May 16, 2013 at 1:04 pm
    Hi Rory,

    Can you share your host file?

    Thanks,

    JM


    2013/5/16 Rory Marland <dj.embed@gmail.com>
    Hi,

    I have tried to running CM4 to install everything onto 6 nodes but get
    heatbeat failed messages along with other install failure messages.
    Please find attached my errors and some more info.

    There is no firewalling on our LAN.

    all done
    closing logging file descriptor
    IOError: Port 9000 not free on 'cloudera-node2'

    [12/Feb/2013 10:38:31 +0000] 23655 MainThread _cplogging ERROR
    [12/Feb/2013:10:38:31] ENGINE Shutting down due to error in start listener:
    Traceback (most recent call last):
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/process/wspbus.py",
    line 235, in start
    self.publish('start')
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/process/wspbus.py",
    line 215, in publish
    raise exc
    ChannelFailures: IOError("Port 9000 not free on 'cloudera-node2'",)

    [12/Feb/2013 10:38:31 +0000] 23655 MainThread _cplogging INFO
    [12/Feb/2013:10:38:31] ENGINE Bus STOPPING
    [12/Feb/2013 10:38:31 +0000] 23655 MainThread _cplogging INFO
    [12/Feb/2013:10:38:31] ENGINE HTTP Server
    cherrypy._cpwsgi_server.CPWSGIServer(('cloudera-node2', 9000)) already shut
    down
    [12/Feb/2013 10:38:31 +0000] 23655 MainThread _cplogging INFO
    [12/Feb/2013:10:38:31] ENGINE Stopped thread '_TimeoutMonitor'.
    [12/Feb/2013 10:38:31 +0000] 23655 MainThread _cplogging INFO
    [12/Feb/2013:10:38:31] ENGINE Bus STOPPED
    [12/Feb/2013 10:38:31 +0000] 23655 MainThread _cplogging INFO
    [12/Feb/2013:10:38:31] ENGINE Bus EXITING
    [12/Feb/2013 10:38:31 +0000] 23655 MainThread _cplogging INFO
    [12/Feb/2013:10:38:31] ENGINE Bus EXITED
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO No command
    line vars
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Agent starting
    as pid 24672 user root(0) group root(0).
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Re-using
    pre-existing directory: /var/run/cloudera-scm-agent
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread downloader INFO
    Downloader path: /opt/cloudera/parcel-cache
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread parcel_cache INFO Using
    /opt/cloudera/parcel-cache for parcel cache
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Re-using
    pre-existing directory: /var/run/cloudera-scm-agent/cgroups
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread cgroups INFO Reusing
    /var/run/cloudera-scm-agent/cgroups/memory
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread cgroups INFO Reusing
    /var/run/cloudera-scm-agent/cgroups/cpu
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread cgroups INFO Reusing
    /var/run/cloudera-scm-agent/cgroups/blkio
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Found cgroups
    capabilities: {'has_memory': True, 'default_memory_limit_in_bytes':
    9223372036854775807, 'writable_cgroup_dot_procs': True, 'has_cpu': True,
    'default_blkio_weight': 1000, 'default_cpu_shares': 1024, 'has_blkio': True}
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Setting up
    supervisord event monitor.
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread filesystem_map INFO
    Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs']
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread filesystem_map INFO Using
    timeout of 2.000000
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread filesystem_map INFO Using
    join timeout of 0.100000
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread filesystem_map INFO Using
    tolerance of 60.000000
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Using
    metrics_url_timeout_seconds of 30.000000
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Using
    task_metrics_timeout_seconds of 5.000000
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Using
    max_collection_wait_seconds of 10.000000
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread metrics INFO Importing
    tasktracker metric schema from file
    /usr/lib64/cmf/agent/src/cmf/monitor/tasktracker/schema.json
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread throttling_logger
    ERROR Failed to collect disk stats for device 'mapper/VolGroup-lv_root'
    Traceback (most recent call last):
    File "/usr/lib64/cmf/agent/src/cmf/monitor/host/disk_device.py", line
    54, in collect_update
    stat_file = open(self._stat_path)
    IOError: [Errno 2] No such file or directory:
    '/sys/block/mapper/VolGroup-lv_root/stat'
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread disk_devices WARNING
    Failed to add DiskDevice for disk='mapper/VolGroup-lv_root'
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread __init__ INFO Importing
    metric schema from file /usr/lib64/cmf/agent/src/cmf/monitor/schema.json
    [12/Feb/2013 10:46:14 +0000] 24672 MainThread agent INFO Supervised
    processes will inherit the following environment: {'CMF_CONF_DIR':
    '/etc/cloudera-scm-agent', 'CMF_AGENT_HIVE_DEFAULT_XML':
    '/etc/hive/conf.dist/hive-default.xml', 'CMF_AGENT_MGMT_HOME':
    '/usr/share/cmf', 'CMF_AGENT_CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs',
    'PATH': '/sbin:/usr/sbin:/bin:/usr/bin', 'CMF_AGENT_CDH_FLUME_HOME':
    '/usr/lib/flume-ng', 'CMF_AGENT_CDH_HBASE_HOME': '/usr/lib/hbase', 'LANG':
    'en_US.UTF-8', 'TERM': 'vt100', 'CLOUDERA_ORACLE_CONNECTOR_JAR':
    '/usr/share/java/oracle-connector-java.jar', 'CMF_AGENT_CDH_HUE_HOME':
    '/usr/share/hue', 'SHLVL': '1', 'CMF_AGENT_TOMCAT_HOME':
    '/usr/lib/bigtop-tomcat', 'CLOUDERA_POSTGRESQL_JDBC_JAR':
    '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar',
    'CMF_AGENT_CDH_PIG_HOME': '/usr/lib/pig', 'CMF_AGENT_CDH_HIVE_HOME':
    '/usr/lib/hive', 'CLOUDERA_MYSQL_CONNECTOR_JAR':
    '/usr/share/java/mysql-connector-java.jar', 'CMF_AGENT_CDH_YARN_HOME':
    '/usr/lib/hadoop-yarn', 'CMF_AGENT_CDH_HADOOP_BIN': '/usr/bin/hadoop',
    'CMF_AGENT_CDH_OOZIE_HOME': '/usr/lib/oozie',
    'CMF_AGENT_CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper',
    'CMF_AGENT_CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CMF_AGENT_JSVC_HOME':
    '/usr/libexec/bigtop-utils', 'CMF_AGENT_CDH_HDFS_HOME':
    '/usr/lib/hadoop-hdfs', 'CMF_AGENT_CDH_MR1_HOME':
    '/usr/lib/hadoop-0.20-mapreduce', 'CMF_AGENT_CDH_MR2_HOME':
    '/usr/lib/hadoop-mapreduce', 'PWD': '/', 'CMF_AGENT_CDH_HADOOP_HOME':
    '/usr/lib/hadoop', 'CMF_AGENT_CDH_IMPALA_HOME': '/usr/lib/impala'}
    [12/Feb/2013 10:46:14 +0000] 24672 MainThread agent INFO Supervised
    processes will add the following their environment environment:
    {'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'CMF_PACKAGE_DIR':
    '/usr/lib64/cmf/service', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME':
    '/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CDH_YARN_HOME':
    '/usr/lib/hadoop-yarn', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs',
    'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE
    HDFS_DFS_DIR_NOT_EMPTY', 'CLOUDERA_ORACLE_CONNECTOR_JAR':
    '/usr/share/java/oracle-connector-java.jar', 'CDH_HIVE_HOME':
    '/usr/lib/hive', 'JSVC_HOME': '/usr/libexec/bigtop-utils',
    'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml',
    'CLOUDERA_POSTGRESQL_JDBC_JAR':
    '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_HBASE_HOME':
    '/usr/lib/hbase', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_ZOOKEEPER_HOME':
    '/usr/lib/zookeeper', 'CDH_HUE_HOME': '/usr/share/hue',
    'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar',
    'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_PIG_HOME':
    '/usr/lib/pig', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs',
    'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'TOMCAT_HOME':
    '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'}
    [12/Feb/2013 10:46:14 +0000] 24672 MainThread agent INFO Re-using
    pre-existing directory: /var/run/cloudera-scm-agent/process
    [12/Feb/2013 10:46:14 +0000] 24672 MainThread agent INFO Re-using
    pre-existing directory: /var/run/cloudera-scm-agent/supervisor
    [12/Feb/2013 10:46:14 +0000] 24672 MainThread agent INFO Re-using
    pre-existing directory: /var/run/cloudera-scm-agent/supervisor/include
    [12/Feb/2013 10:46:14 +0000] 24672 MainThread agent INFO Connecting to
    previous supervisor: agent-8767-1360595961.
    [12/Feb/2013 10:46:14 +0000] 24672 MainThread _cplogging INFO
    [12/Feb/2013:10:46:14] ENGINE Bus STARTING
    [12/Feb/2013 10:46:14 +0000] 24672 MainThread _cplogging INFO
    [12/Feb/2013:10:46:14] ENGINE Started monitor thread '_TimeoutMonitor'.

    IP Tables are OFF
    SELinux is DISABLED

    Netstat readout

    Active Internet connections (servers and established)
    Proto Recv-Q Send-Q Local Address Foreign
    Address State PID/Program name
    tcp 0 0 cloudera-node1:9000
    *:* LISTEN 17258/python
    tcp 0 0 *:7432
    *:* LISTEN 1798/postgres
    tcp 0 0 localhost:9001
    *:* LISTEN 8945/python2.6
    tcp 0 0 *:7180
    *:* LISTEN 1826/java
    tcp 0 0 *:7182
    *:* LISTEN 1826/java
    tcp 0 0 *:111
    *:* LISTEN 1426/rpcbind
    tcp 0 0 *:22
    *:* LISTEN
    tcp 0 0 *:54102
    *:* LISTEN
    tcp 0 0 localhost:36186
    localhost:9001 ESTABLISHED
    tcp 0 0 cloudera-node1:38815
    cloudera-node3:22 ESTABLISHED
    tcp 0 0 localhost:7432
    localhost:58248 ESTABLISHED
    tcp 0 0 cloudera-node1:7180 webvpn.domain.co.uk:16971
    ESTABLISHED
    tcp 0 0 localhost:58245
    localhost:7432 ESTABLISHED
    tcp 0 0 localhost:58247
    localhost:7432 ESTABLISHED
    tcp 0 0 cloudera-node1:39077
    cloudera-node1:7182 ESTABLISHED
    tcp 0 0 localhost:7432
    localhost:58246 ESTABLISHED
    tcp 0 0 cloudera-node1:22 webvpn.domain.co.uk:16062
    ESTABLISHED
    tcp 0 0 cloudera-node1:43521
    cloudera-node6:22 ESTABLISHED
    tcp 0 0 localhost:58246
    localhost:7432 ESTABLISHED
    tcp 0 0 localhost:58248
    localhost:7432 ESTABLISHED
    tcp 0 0 cloudera-node1:49106
    cloudera-node5:22 ESTABLISHED
    tcp 0 0 cloudera-node1:49705
    cloudera-node4:22 ESTABLISHED
    tcp 0 0 localhost:7432
    localhost:58244 ESTABLISHED
    tcp 0 0 localhost:7432
    localhost:58247 ESTABLISHED
    tcp 0 0 localhost:7432
    localhost:58245 ESTABLISHED
    tcp 0 0 localhost:9001
    localhost:36186 ESTABLISHED
    tcp 0 0 cloudera-node1:7182
    cloudera-node1:39077 ESTABLISHED
    tcp 0 0 localhost:58244
    localhost:7432 ESTABLISHED
    tcp 0 0 *:7432
    *:* LISTEN
    tcp 0 0 *:33514
    *:* LISTEN
    tcp 0 0 *:111
    *:* LISTEN
    tcp 0 0 *:22
    *:* LISTEN

    Any thoughts on how to resolve this?
  • Philip Zeyliger at May 16, 2013 at 2:15 pm
    ChannelFailures: IOError("Port 9000 not free on 'cloudera-node2'",)
    That suggests that some other process is using port 9000 on the machine,
    which is the port the agent tries to bind to. (9000 and 9001 both, to be
    precise.)

    Try killing whatever is listening on 9000/9001. You can re-configure the
    ports as well in /etc/cloudera-scm-agent/config.ini, if need be.

    -- Philip

    On Thu, May 16, 2013 at 6:04 AM, Jean-Marc Spaggiari wrote:

    Hi Rory,

    Can you share your host file?

    Thanks,

    JM


    2013/5/16 Rory Marland <dj.embed@gmail.com>
    Hi,

    I have tried to running CM4 to install everything onto 6 nodes but get
    heatbeat failed messages along with other install failure messages.
    Please find attached my errors and some more info.

    There is no firewalling on our LAN.

    all done
    closing logging file descriptor
    IOError: Port 9000 not free on 'cloudera-node2'

    [12/Feb/2013 10:38:31 +0000] 23655 MainThread _cplogging ERROR
    [12/Feb/2013:10:38:31] ENGINE Shutting down due to error in start listener:
    Traceback (most recent call last):
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/process/wspbus.py",
    line 235, in start
    self.publish('start')
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/process/wspbus.py",
    line 215, in publish
    raise exc
    ChannelFailures: IOError("Port 9000 not free on 'cloudera-node2'",)

    [12/Feb/2013 10:38:31 +0000] 23655 MainThread _cplogging INFO
    [12/Feb/2013:10:38:31] ENGINE Bus STOPPING
    [12/Feb/2013 10:38:31 +0000] 23655 MainThread _cplogging INFO
    [12/Feb/2013:10:38:31] ENGINE HTTP Server
    cherrypy._cpwsgi_server.CPWSGIServer(('cloudera-node2', 9000)) already shut
    down
    [12/Feb/2013 10:38:31 +0000] 23655 MainThread _cplogging INFO
    [12/Feb/2013:10:38:31] ENGINE Stopped thread '_TimeoutMonitor'.
    [12/Feb/2013 10:38:31 +0000] 23655 MainThread _cplogging INFO
    [12/Feb/2013:10:38:31] ENGINE Bus STOPPED
    [12/Feb/2013 10:38:31 +0000] 23655 MainThread _cplogging INFO
    [12/Feb/2013:10:38:31] ENGINE Bus EXITING
    [12/Feb/2013 10:38:31 +0000] 23655 MainThread _cplogging INFO
    [12/Feb/2013:10:38:31] ENGINE Bus EXITED
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO No command
    line vars
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Agent
    starting as pid 24672 user root(0) group root(0).
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Re-using
    pre-existing directory: /var/run/cloudera-scm-agent
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread downloader INFO
    Downloader path: /opt/cloudera/parcel-cache
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread parcel_cache INFO Using
    /opt/cloudera/parcel-cache for parcel cache
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Re-using
    pre-existing directory: /var/run/cloudera-scm-agent/cgroups
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread cgroups INFO Reusing
    /var/run/cloudera-scm-agent/cgroups/memory
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread cgroups INFO Reusing
    /var/run/cloudera-scm-agent/cgroups/cpu
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread cgroups INFO Reusing
    /var/run/cloudera-scm-agent/cgroups/blkio
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Found cgroups
    capabilities: {'has_memory': True, 'default_memory_limit_in_bytes':
    9223372036854775807, 'writable_cgroup_dot_procs': True, 'has_cpu': True,
    'default_blkio_weight': 1000, 'default_cpu_shares': 1024, 'has_blkio': True}
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Setting up
    supervisord event monitor.
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread filesystem_map INFO
    Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs']
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread filesystem_map INFO
    Using timeout of 2.000000
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread filesystem_map INFO
    Using join timeout of 0.100000
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread filesystem_map INFO
    Using tolerance of 60.000000
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Using
    metrics_url_timeout_seconds of 30.000000
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Using
    task_metrics_timeout_seconds of 5.000000
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Using
    max_collection_wait_seconds of 10.000000
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread metrics INFO Importing
    tasktracker metric schema from file
    /usr/lib64/cmf/agent/src/cmf/monitor/tasktracker/schema.json
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread throttling_logger
    ERROR Failed to collect disk stats for device 'mapper/VolGroup-lv_root'
    Traceback (most recent call last):
    File "/usr/lib64/cmf/agent/src/cmf/monitor/host/disk_device.py", line
    54, in collect_update
    stat_file = open(self._stat_path)
    IOError: [Errno 2] No such file or directory:
    '/sys/block/mapper/VolGroup-lv_root/stat'
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread disk_devices WARNING
    Failed to add DiskDevice for disk='mapper/VolGroup-lv_root'
    [12/Feb/2013 10:46:12 +0000] 24672 MainThread __init__ INFO Importing
    metric schema from file /usr/lib64/cmf/agent/src/cmf/monitor/schema.json
    [12/Feb/2013 10:46:14 +0000] 24672 MainThread agent INFO Supervised
    processes will inherit the following environment: {'CMF_CONF_DIR':
    '/etc/cloudera-scm-agent', 'CMF_AGENT_HIVE_DEFAULT_XML':
    '/etc/hive/conf.dist/hive-default.xml', 'CMF_AGENT_MGMT_HOME':
    '/usr/share/cmf', 'CMF_AGENT_CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs',
    'PATH': '/sbin:/usr/sbin:/bin:/usr/bin', 'CMF_AGENT_CDH_FLUME_HOME':
    '/usr/lib/flume-ng', 'CMF_AGENT_CDH_HBASE_HOME': '/usr/lib/hbase', 'LANG':
    'en_US.UTF-8', 'TERM': 'vt100', 'CLOUDERA_ORACLE_CONNECTOR_JAR':
    '/usr/share/java/oracle-connector-java.jar', 'CMF_AGENT_CDH_HUE_HOME':
    '/usr/share/hue', 'SHLVL': '1', 'CMF_AGENT_TOMCAT_HOME':
    '/usr/lib/bigtop-tomcat', 'CLOUDERA_POSTGRESQL_JDBC_JAR':
    '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar',
    'CMF_AGENT_CDH_PIG_HOME': '/usr/lib/pig', 'CMF_AGENT_CDH_HIVE_HOME':
    '/usr/lib/hive', 'CLOUDERA_MYSQL_CONNECTOR_JAR':
    '/usr/share/java/mysql-connector-java.jar', 'CMF_AGENT_CDH_YARN_HOME':
    '/usr/lib/hadoop-yarn', 'CMF_AGENT_CDH_HADOOP_BIN': '/usr/bin/hadoop',
    'CMF_AGENT_CDH_OOZIE_HOME': '/usr/lib/oozie',
    'CMF_AGENT_CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper',
    'CMF_AGENT_CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CMF_AGENT_JSVC_HOME':
    '/usr/libexec/bigtop-utils', 'CMF_AGENT_CDH_HDFS_HOME':
    '/usr/lib/hadoop-hdfs', 'CMF_AGENT_CDH_MR1_HOME':
    '/usr/lib/hadoop-0.20-mapreduce', 'CMF_AGENT_CDH_MR2_HOME':
    '/usr/lib/hadoop-mapreduce', 'PWD': '/', 'CMF_AGENT_CDH_HADOOP_HOME':
    '/usr/lib/hadoop', 'CMF_AGENT_CDH_IMPALA_HOME': '/usr/lib/impala'}
    [12/Feb/2013 10:46:14 +0000] 24672 MainThread agent INFO Supervised
    processes will add the following their environment environment:
    {'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'CMF_PACKAGE_DIR':
    '/usr/lib64/cmf/service', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME':
    '/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CDH_YARN_HOME':
    '/usr/lib/hadoop-yarn', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs',
    'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE
    HDFS_DFS_DIR_NOT_EMPTY', 'CLOUDERA_ORACLE_CONNECTOR_JAR':
    '/usr/share/java/oracle-connector-java.jar', 'CDH_HIVE_HOME':
    '/usr/lib/hive', 'JSVC_HOME': '/usr/libexec/bigtop-utils',
    'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml',
    'CLOUDERA_POSTGRESQL_JDBC_JAR':
    '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_HBASE_HOME':
    '/usr/lib/hbase', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_ZOOKEEPER_HOME':
    '/usr/lib/zookeeper', 'CDH_HUE_HOME': '/usr/share/hue',
    'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar',
    'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_PIG_HOME':
    '/usr/lib/pig', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs',
    'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'TOMCAT_HOME':
    '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'}
    [12/Feb/2013 10:46:14 +0000] 24672 MainThread agent INFO Re-using
    pre-existing directory: /var/run/cloudera-scm-agent/process
    [12/Feb/2013 10:46:14 +0000] 24672 MainThread agent INFO Re-using
    pre-existing directory: /var/run/cloudera-scm-agent/supervisor
    [12/Feb/2013 10:46:14 +0000] 24672 MainThread agent INFO Re-using
    pre-existing directory: /var/run/cloudera-scm-agent/supervisor/include
    [12/Feb/2013 10:46:14 +0000] 24672 MainThread agent INFO Connecting to
    previous supervisor: agent-8767-1360595961.
    [12/Feb/2013 10:46:14 +0000] 24672 MainThread _cplogging INFO
    [12/Feb/2013:10:46:14] ENGINE Bus STARTING
    [12/Feb/2013 10:46:14 +0000] 24672 MainThread _cplogging INFO
    [12/Feb/2013:10:46:14] ENGINE Started monitor thread '_TimeoutMonitor'.

    IP Tables are OFF
    SELinux is DISABLED

    Netstat readout

    Active Internet connections (servers and established)
    Proto Recv-Q Send-Q Local Address Foreign
    Address State PID/Program name
    tcp 0 0 cloudera-node1:9000
    *:* LISTEN 17258/python
    tcp 0 0 *:7432
    *:* LISTEN 1798/postgres
    tcp 0 0 localhost:9001
    *:* LISTEN 8945/python2.6
    tcp 0 0 *:7180
    *:* LISTEN 1826/java
    tcp 0 0 *:7182
    *:* LISTEN 1826/java
    tcp 0 0 *:111
    *:* LISTEN 1426/rpcbind
    tcp 0 0 *:22
    *:* LISTEN
    tcp 0 0 *:54102
    *:* LISTEN
    tcp 0 0 localhost:36186
    localhost:9001 ESTABLISHED
    tcp 0 0 cloudera-node1:38815
    cloudera-node3:22 ESTABLISHED
    tcp 0 0 localhost:7432
    localhost:58248 ESTABLISHED
    tcp 0 0 cloudera-node1:7180
    webvpn.domain.co.uk:16971 ESTABLISHED
    tcp 0 0 localhost:58245
    localhost:7432 ESTABLISHED
    tcp 0 0 localhost:58247
    localhost:7432 ESTABLISHED
    tcp 0 0 cloudera-node1:39077
    cloudera-node1:7182 ESTABLISHED
    tcp 0 0 localhost:7432
    localhost:58246 ESTABLISHED
    tcp 0 0 cloudera-node1:22
    webvpn.domain.co.uk:16062 ESTABLISHED
    tcp 0 0 cloudera-node1:43521
    cloudera-node6:22 ESTABLISHED
    tcp 0 0 localhost:58246
    localhost:7432 ESTABLISHED
    tcp 0 0 localhost:58248
    localhost:7432 ESTABLISHED
    tcp 0 0 cloudera-node1:49106
    cloudera-node5:22 ESTABLISHED
    tcp 0 0 cloudera-node1:49705
    cloudera-node4:22 ESTABLISHED
    tcp 0 0 localhost:7432
    localhost:58244 ESTABLISHED
    tcp 0 0 localhost:7432
    localhost:58247 ESTABLISHED
    tcp 0 0 localhost:7432
    localhost:58245 ESTABLISHED
    tcp 0 0 localhost:9001
    localhost:36186 ESTABLISHED
    tcp 0 0 cloudera-node1:7182
    cloudera-node1:39077 ESTABLISHED
    tcp 0 0 localhost:58244
    localhost:7432 ESTABLISHED
    tcp 0 0 *:7432
    *:* LISTEN
    tcp 0 0 *:33514
    *:* LISTEN
    tcp 0 0 *:111
    *:* LISTEN
    tcp 0 0 *:22
    *:* LISTEN

    Any thoughts on how to resolve this?

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedMay 16, '13 at 10:46a
activeMay 16, '13 at 2:15p
posts3
users3
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase