Hello, I am trying to set up a Hadoop/Hbase cluster on EC2. I have to use
Hadoop 1.0.4 and HBase 0.94, so I followed this guide:
http://rogueleaderr.com/post/32768181371/set-up-and-run-a-fully-distributed-hadoop-hbase-cluster
I realize the instructions are depreciated, but I know the latest Cloudera
CDH doesn't run Hadoop 1.0.4.
I installed the correct versions of Hadoop and Hbase from the Apache
archives and set up the cluster using Whirr and all went well. Haddop and
Hbase are running - however, the configurations and start/stop scripts
weren't working well, so I thought installing the Cloudera Manager would
help in that regard.
Anyway, I give the install process the IPs of my boxes (I have 1 master and
3 slaves) and confirming that they were reachable by pings, I started the
installation process- all was going well until the process was waiting for
a heartbeat and each of my boxes failed to give one and then the install
failed.
I figured maybe the internal ip's of the EC2 instances was the problem,
changed all the hostnames to their FQDNs and still the same issue happened.
My log file is as follows:
31/Jul/2013 17:53:27 +0000] 18298 MainThread agent INFO No
command line vars
[31/Jul/2013 17:53:27 +0000] 18298 MainThread agent INFO Missing
database jar: /usr/share/java/mysql-connector-java.jar (normal, if you're
not using this database type)
[31/Jul/2013 17:53:27 +0000] 18298 MainThread agent INFO Missing
database jar: /usr/share/java/oracle-connector-java.jar (normal, if you're
not using this database type)
[31/Jul/2013 17:53:27 +0000] 18298 MainThread agent INFO Found
database jar: /usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar
[31/Jul/2013 17:53:27 +0000] 18298 MainThread agent INFO Agent
starting as pid 18298 user root(0) group root(0).
[31/Jul/2013 17:53:27 +0000] 18298 MainThread agent INFO Created
/run/cloudera-scm-agent
[31/Jul/2013 17:53:27 +0000] 18298 MainThread agent INFO
Chmod'ing /run/cloudera-scm-agent to 0751
[31/Jul/2013 17:53:27 +0000] 18298 MainThread agent INFO Agent
is running on a public cloud. Public hostname:
ec2-54-242-227-124.compute-1.amazonaws.com
[31/Jul/2013 17:53:27 +0000] 18298 MainThread agent INFO Created
/run/cloudera-scm-agent/cgroups
[31/Jul/2013 17:53:27 +0000] 18298 MainThread agent INFO
Chmod'ing /run/cloudera-scm-agent/cgroups to 0751
[31/Jul/2013 17:53:27 +0000] 18298 MainThread cgroups INFO Created
/run/cloudera-scm-agent/cgroups/memory
[31/Jul/2013 17:53:27 +0000] 18298 MainThread cgroups INFO Created
/run/cloudera-scm-agent/cgroups/cpu
[31/Jul/2013 17:53:27 +0000] 18298 MainThread cgroups INFO Created
/run/cloudera-scm-agent/cgroups/cpuacct
[31/Jul/2013 17:53:27 +0000] 18298 MainThread cgroups INFO Created
/run/cloudera-scm-agent/cgroups/blkio
[31/Jul/2013 17:53:27 +0000] 18298 MainThread agent INFO Found
cgroups capabilities: {'has_memory': True, 'default_memory_limit_in_bytes':
9223372036854775807, 'default_memory_soft_limit_in_bytes':
9223372036854775807, 'writable_cgroup_dot_procs': True,
'default_cpu_rt_runtime_us': 950000, 'has_cpu': True,
'default_blkio_weight': 1000, 'default_cpu_shares': 1024, 'has_cpuacct':
True, 'has_blkio': True}
[31/Jul/2013 17:53:27 +0000] 18298 MainThread agent INFO Setting
up supervisord event monitor.
[31/Jul/2013 17:53:27 +0000] 18298 MainThread filesystem_map INFO
Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs']
[31/Jul/2013 17:53:27 +0000] 18298 MainThread filesystem_map INFO Using
timeout of 2.000000
[31/Jul/2013 17:53:27 +0000] 18298 MainThread filesystem_map INFO Using
join timeout of 0.100000
[31/Jul/2013 17:53:27 +0000] 18298 MainThread filesystem_map INFO Using
tolerance of 60.000000
[31/Jul/2013 17:53:27 +0000] 18298 MainThread agent INFO Using
metrics_url_timeout_seconds of 30.000000
[31/Jul/2013 17:53:27 +0000] 18298 MainThread agent INFO Using
task_metrics_timeout_seconds of 5.000000
[31/Jul/2013 17:53:27 +0000] 18298 MainThread agent INFO Using
max_collection_wait_seconds of 10.000000
[31/Jul/2013 17:53:27 +0000] 18298 MainThread metrics INFO
Importing tasktracker metric schema from file
/usr/lib/cmf/agent/src/cmf/monitor/tasktracker/schema.json
[31/Jul/2013 17:53:27 +0000] 18298 MainThread dns_names INFO Using
timeout of 2.000000
[31/Jul/2013 17:53:27 +0000] 18298 MainThread __init__ INFO
Importing metric schema from file
/usr/lib/cmf/agent/src/cmf/monitor/schema.json
[31/Jul/2013 17:53:27 +0000] 18298 MainThread agent INFO
Supervised processes will inherit the following environment: {'LANG':
'en_US.UTF-8', 'TERM': 'vt100', 'CMF_CONF_DIR': '/etc/cloudera-scm-agent',
'SHLVL': '1', 'PWD': '/', 'PATH':
'/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games'}
[31/Jul/2013 17:53:27 +0000] 18298 MainThread agent INFO
Supervised processes will add the following to their environment:
{'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_MR2_HOME':
'/usr/lib/hadoop-mapreduce', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper',
'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/usr/share/cmf',
'CDH_IMPALA_HOME': '/usr/lib/impala', 'CLOUDERA_MYSQL_CONNECTOR_JAR':
'/usr/share/java/mysql-connector-java.jar', 'CDH_YARN_HOME':
'/usr/lib/hadoop-yarn', 'CMF_PACKAGE_DIR': '/usr/lib/cmf/service',
'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'CDH_SOLR_HOME': '/usr/lib/solr',
'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2',
'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE
HDFS_DFS_DIR_NOT_EMPTY JOBTRACKER_IN_STANDBY_MODE', 'CDH_MR1_HOME':
'/usr/lib/hadoop-0.20-mapreduce', 'CLOUDERA_ORACLE_CONNECTOR_JAR':
'/usr/share/java/oracle-connector-java.jar', 'CDH_HUE_HOME':
'/usr/share/hue', 'CDH_HIVE_HOME': '/usr/lib/hive', 'CDH_HTTPFS_HOME':
'/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop',
'JSVC_HOME': '/usr/libexec/bigtop-utils', 'HIVE_DEFAULT_XML':
'/etc/hive/conf.dist/hive-default.xml', 'CDH_FLUME_HOME':
'/usr/lib/flume-ng', 'CLOUDERA_POSTGRESQL_JDBC_JAR':
'/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'TOMCAT_HOME':
'/usr/lib/bigtop-tomcat', 'CDH_HBASE_HOME': '/usr/lib/hbase',
'CDH_HCAT_HOME': '/usr/lib/hcatalog'}
[31/Jul/2013 17:53:27 +0000] 18298 MainThread agent INFO To
override these variables, use /etc/cloudera-scm-agent/config.ini.
Environment variables for CDH locations are not used when CDH is installed
from parcels.
[31/Jul/2013 17:53:27 +0000] 18298 MainThread agent INFO Created
/run/cloudera-scm-agent/process
[31/Jul/2013 17:53:27 +0000] 18298 MainThread agent INFO
Chmod'ing /run/cloudera-scm-agent/process to 0751
[31/Jul/2013 17:53:27 +0000] 18298 MainThread agent INFO Created
/run/cloudera-scm-agent/supervisor
[31/Jul/2013 17:53:27 +0000] 18298 MainThread agent INFO
Chmod'ing /run/cloudera-scm-agent/supervisor to 0751
[31/Jul/2013 17:53:27 +0000] 18298 MainThread agent INFO Created
/run/cloudera-scm-agent/supervisor/include
[31/Jul/2013 17:53:27 +0000] 18298 MainThread agent INFO
Chmod'ing /run/cloudera-scm-agent/supervisor/include to 0751
[31/Jul/2013 17:53:27 +0000] 18298 MainThread agent INFO Failed
to connect to previous supervisor.
[31/Jul/2013 17:53:29 +0000] 18298 MainThread agent INFO Trying
to connect to newly launched supervisor (Attempt 1)
[31/Jul/2013 17:53:29 +0000] 18298 MainThread agent INFO
Successfully connected to supervisor
[31/Jul/2013 17:53:29 +0000] 18298 MainThread _cplogging INFO
[31/Jul/2013:17:53:29] ENGINE Bus STARTING
[31/Jul/2013 17:53:29 +0000] 18298 MainThread _cplogging INFO
[31/Jul/2013:17:53:29] ENGINE Started monitor thread '_TimeoutMonitor'.
[31/Jul/2013 17:53:29 +0000] 18298 MainThread _cplogging INFO
[31/Jul/2013:17:53:29] ENGINE Serving on ip-10-77-74-189.ec2.internal:9000
[31/Jul/2013 17:53:29 +0000] 18298 MainThread _cplogging INFO
[31/Jul/2013:17:53:29] ENGINE Bus STARTED
[31/Jul/2013 17:53:29 +0000] 18298 MainThread __init__ INFO New
monitor: (<cmf.monitor.host.hostmonitor object@0x7f4e98496790>,)
[31/Jul/2013 17:53:29 +0000] 18298 MainThread agent WARNING Setting
default socket timeout to 30!
[31/Jul/2013 17:53:29 +0000] 18298 MonitorDaemon-Scheduler __init__
INFO Monitor ready to report: (<cmf.monitor.host.HostMonitor object at
0x7f4e98496790>,)
[31/Jul/2013 17:53:29 +0000] 18298 MainThread agent INFO Created
/opt/cloudera/parcels
[31/Jul/2013 17:53:29 +0000] 18298 MainThread agent INFO
Chowning /opt/cloudera/parcels to root (0) root (0)
[31/Jul/2013 17:53:29 +0000] 18298 MainThread agent INFO
Chmod'ing /opt/cloudera/parcels to 0755
[31/Jul/2013 17:53:29 +0000] 18298 MainThread parcel INFO Agent
creates users/groups and applies file permissions: True
[31/Jul/2013 17:53:29 +0000] 18298 MainThread agent INFO Created
/opt/cloudera/parcel-cache
[31/Jul/2013 17:53:29 +0000] 18298 MainThread agent INFO
Chowning /opt/cloudera/parcel-cache to root (0) root (0)
[31/Jul/2013 17:53:29 +0000] 18298 MainThread agent INFO
Chmod'ing /opt/cloudera/parcel-cache to 0755
[31/Jul/2013 17:53:29 +0000] 18298 MainThread downloader INFO
Downloader path: /opt/cloudera/parcel-cache
[31/Jul/2013 17:53:29 +0000] 18298 MainThread parcel_cache INFO Using
/opt/cloudera/parcel-cache for parcel cache
[31/Jul/2013 17:53:34 +0000] 18298 Monitor-HostMonitor network_interfaces
INFO NIC iface eth0 doesn't support ETHTOOL (95)
[31/Jul/2013 18:00:43 +0000] 18298 MainThread agent INFO
Stopping agent...
[31/Jul/2013 18:00:43 +0000] 18298 MainThread agent INFO No
cgrouped processes running; unmounting any cgroup roots
[31/Jul/2013 18:00:43 +0000] 18298 MainThread agent INFO No
processes are being managed; Supervisor will shutdown.
[31/Jul/2013 18:00:43 +0000] 18298 MainThread agent INFO
Shutting down supervisord, pid 18318
[31/Jul/2013 18:00:43 +0000] 18298 MainThread agent INFO
Successfully shutdown supervisord, pid 18318
[31/Jul/2013 18:00:43 +0000] 18298 MainThread _cplogging INFO
[31/Jul/2013:18:00:43] ENGINE Bus STOPPING
[31/Jul/2013 18:00:43 +0000] 18298 MainThread _cplogging INFO
[31/Jul/2013:18:00:43] ENGINE HTTP Server
cherrypy._cpwsgi_server.CPWSGIServer(('ip-10-77-74-189.ec2.internal',
9000)) shut down
[31/Jul/2013 18:00:43 +0000] 18298 MainThread _cplogging INFO
[31/Jul/2013:18:00:43] ENGINE Stopped thread '_TimeoutMonitor'.
[31/Jul/2013 18:00:43 +0000] 18298 MainThread _cplogging INFO
[31/Jul/2013:18:00:43] ENGINE Bus STOPPED
[31/Jul/2013 18:00:43 +0000] 18298 MainThread _cplogging INFO
[31/Jul/2013:18:00:43] ENGINE Bus STOPPING
[31/Jul/2013 18:00:43 +0000] 18298 MainThread _cplogging INFO
[31/Jul/2013:18:00:43] ENGINE HTTP Server
cherrypy._cpwsgi_server.CPWSGIServer(('ip-10-77-74-189.ec2.internal',
9000)) already shut down
[31/Jul/2013 18:00:43 +0000] 18298 MainThread _cplogging INFO
[31/Jul/2013:18:00:43] ENGINE No thread running for None.
[31/Jul/2013 18:00:43 +0000] 18298 MainThread _cplogging INFO
[31/Jul/2013:18:00:43] ENGINE Bus STOPPED
[31/Jul/2013 18:00:43 +0000] 18298 MainThread _cplogging INFO
[31/Jul/2013:18:00:43] ENGINE Bus EXITING
[31/Jul/2013 18:00:43 +0000] 18298 MainThread _cplogging INFO
[31/Jul/2013:18:00:43] ENGINE Bus EXITED
[31/Jul/2013 18:00:43 +0000] 18298 MainThread agent INFO Agent
exiting; caught signal 15
[31/Jul/2013 18:00:43 +0000] 19950 MainThread agent INFO No
command line vars
[31/Jul/2013 18:00:43 +0000] 19950 MainThread agent INFO Missing
database jar: /usr/share/java/mysql-connector-java.jar (normal, if you're
not using this database type)
[31/Jul/2013 18:00:43 +0000] 19950 MainThread agent INFO Missing
database jar: /usr/share/java/oracle-connector-java.jar (normal, if you're
not using this database type)
[31/Jul/2013 18:00:43 +0000] 19950 MainThread agent INFO Found
database jar: /usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar
[31/Jul/2013 18:00:43 +0000] 19950 MainThread agent INFO Agent
starting as pid 19950 user root(0) group root(0).
[31/Jul/2013 18:00:43 +0000] 19950 MainThread agent INFO
Re-using pre-existing directory: /run/cloudera-scm-agent
[31/Jul/2013 18:00:43 +0000] 19950 MainThread agent INFO Agent
is running on a public cloud. Public hostname:
ec2-54-242-227-124.compute-1.amazonaws.com
[31/Jul/2013 18:00:43 +0000] 19950 MainThread agent INFO
Re-using pre-existing directory: /run/cloudera-scm-agent/cgroups
[31/Jul/2013 18:00:44 +0000] 19950 MainThread cgroups INFO Reusing
/run/cloudera-scm-agent/cgroups/memory
[31/Jul/2013 18:00:44 +0000] 19950 MainThread cgroups INFO Reusing
/run/cloudera-scm-agent/cgroups/cpu
[31/Jul/2013 18:00:44 +0000] 19950 MainThread cgroups INFO Reusing
/run/cloudera-scm-agent/cgroups/cpuacct
[31/Jul/2013 18:00:44 +0000] 19950 MainThread cgroups INFO Reusing
/run/cloudera-scm-agent/cgroups/blkio
[31/Jul/2013 18:00:44 +0000] 19950 MainThread agent INFO Found
cgroups capabilities: {'has_memory': True, 'default_memory_limit_in_bytes':
9223372036854775807, 'default_memory_soft_limit_in_bytes':
9223372036854775807, 'writable_cgroup_dot_procs': True,
'default_cpu_rt_runtime_us': 950000, 'has_cpu': True,
'default_blkio_weight': 1000, 'default_cpu_shares': 1024, 'has_cpuacct':
True, 'has_blkio': True}
[31/Jul/2013 18:00:44 +0000] 19950 MainThread agent INFO Setting
up supervisord event monitor.
[31/Jul/2013 18:00:44 +0000] 19950 MainThread filesystem_map INFO
Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs']
[31/Jul/2013 18:00:44 +0000] 19950 MainThread filesystem_map INFO Using
timeout of 2.000000
[31/Jul/2013 18:00:44 +0000] 19950 MainThread filesystem_map INFO Using
join timeout of 0.100000
[31/Jul/2013 18:00:44 +0000] 19950 MainThread filesystem_map INFO Using
tolerance of 60.000000
[31/Jul/2013 18:00:44 +0000] 19950 MainThread agent INFO Using
metrics_url_timeout_seconds of 30.000000
[31/Jul/2013 18:00:44 +0000] 19950 MainThread agent INFO Using
task_metrics_timeout_seconds of 5.000000
[31/Jul/2013 18:00:44 +0000] 19950 MainThread agent INFO Using
max_collection_wait_seconds of 10.000000
[31/Jul/2013 18:00:44 +0000] 19950 MainThread metrics INFO
Importing tasktracker metric schema from file
/usr/lib/cmf/agent/src/cmf/monitor/tasktracker/schema.json
[31/Jul/2013 18:00:44 +0000] 19950 MainThread dns_names INFO Using
timeout of 2.000000
[31/Jul/2013 18:00:44 +0000] 19950 MainThread __init__ INFO
Importing metric schema from file
/usr/lib/cmf/agent/src/cmf/monitor/schema.json
[31/Jul/2013 18:00:44 +0000] 19950 MainThread agent INFO
Supervised processes will inherit the following environment: {'LANG':
'en_US.UTF-8', 'TERM': 'vt100', 'CMF_CONF_DIR': '/etc/cloudera-scm-agent',
'SHLVL': '1', 'PWD': '/', 'PATH':
'/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games'}
[31/Jul/2013 18:00:44 +0000] 19950 MainThread agent INFO
Supervised processes will add the following to their environment:
{'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_MR2_HOME':
'/usr/lib/hadoop-mapreduce', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper',
'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/usr/share/cmf',
'CDH_IMPALA_HOME': '/usr/lib/impala', 'CLOUDERA_MYSQL_CONNECTOR_JAR':
'/usr/share/java/mysql-connector-java.jar', 'CDH_YARN_HOME':
'/usr/lib/hadoop-yarn', 'CMF_PACKAGE_DIR': '/usr/lib/cmf/service',
'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'CDH_SOLR_HOME': '/usr/lib/solr',
'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2',
'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE
HDFS_DFS_DIR_NOT_EMPTY JOBTRACKER_IN_STANDBY_MODE', 'CDH_MR1_HOME':
'/usr/lib/hadoop-0.20-mapreduce', 'CLOUDERA_ORACLE_CONNECTOR_JAR':
'/usr/share/java/oracle-connector-java.jar', 'CDH_HUE_HOME':
'/usr/share/hue', 'CDH_HIVE_HOME': '/usr/lib/hive', 'CDH_HTTPFS_HOME':
'/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop',
'JSVC_HOME': '/usr/libexec/bigtop-utils', 'HIVE_DEFAULT_XML':
'/etc/hive/conf.dist/hive-default.xml', 'CDH_FLUME_HOME':
'/usr/lib/flume-ng', 'CLOUDERA_POSTGRESQL_JDBC_JAR':
'/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'TOMCAT_HOME':
'/usr/lib/bigtop-tomcat', 'CDH_HBASE_HOME': '/usr/lib/hbase',
'CDH_HCAT_HOME': '/usr/lib/hcatalog'}
[31/Jul/2013 18:00:44 +0000] 19950 MainThread agent INFO To
override these variables, use /etc/cloudera-scm-agent/config.ini.
Environment variables for CDH locations are not used when CDH is installed
from parcels.
[31/Jul/2013 18:00:44 +0000] 19950 MainThread agent INFO
Re-using pre-existing directory: /run/cloudera-scm-agent/process
[31/Jul/2013 18:00:44 +0000] 19950 MainThread agent INFO
Re-using pre-existing directory: /run/cloudera-scm-agent/supervisor
[31/Jul/2013 18:00:44 +0000] 19950 MainThread agent INFO
Re-using pre-existing directory: /run/cloudera-scm-agent/supervisor/include
[31/Jul/2013 18:00:44 +0000] 19950 MainThread agent INFO Failed
to connect to previous supervisor.
[31/Jul/2013 18:00:46 +0000] 19950 MainThread agent INFO Trying
to connect to newly launched supervisor (Attempt 1)
[31/Jul/2013 18:00:46 +0000] 19950 MainThread agent INFO
Successfully connected to supervisor
[31/Jul/2013 18:00:46 +0000] 19950 MainThread _cplogging INFO
[31/Jul/2013:18:00:46] ENGINE Bus STARTING
[31/Jul/2013 18:00:46 +0000] 19950 MainThread _cplogging INFO
[31/Jul/2013:18:00:46] ENGINE Started monitor thread '_TimeoutMonitor'.
[31/Jul/2013 18:00:46 +0000] 19950 MainThread _cplogging INFO
[31/Jul/2013:18:00:46] ENGINE Serving on ip-10-77-74-189.ec2.internal:9000
[31/Jul/2013 18:00:46 +0000] 19950 MainThread _cplogging INFO
[31/Jul/2013:18:00:46] ENGINE Bus STARTED
[31/Jul/2013 18:00:46 +0000] 19950 MainThread __init__ INFO New
monitor: (<cmf.monitor.host.hostmonitor object@0x2b86f50>,)
[31/Jul/2013 18:00:46 +0000] 19950 MainThread agent WARNING Setting
default socket timeout to 30!
[31/Jul/2013 18:00:46 +0000] 19950 MonitorDaemon-Scheduler __init__
INFO Monitor ready to report: (<cmf.monitor.host.HostMonitor object at
0x2b86f50>,)
[31/Jul/2013 18:00:46 +0000] 19950 MainThread parcel INFO Agent
creates users/groups and applies file permissions: True
[31/Jul/2013 18:00:46 +0000] 19950 MainThread downloader INFO
Downloader path: /opt/cloudera/parcel-cache
[31/Jul/2013 18:00:46 +0000] 19950 MainThread parcel_cache INFO Using
/opt/cloudera/parcel-cache for parcel cache
[31/Jul/2013 18:00:51 +0000] 19950 Monitor-HostMonitor network_interfaces
INFO NIC iface eth0 doesn't support ETHTOOL (95)
[31/Jul/2013 18:16:32 +0000] 19950 MainThread agent INFO
Stopping agent...
[31/Jul/2013 18:16:32 +0000] 19950 MainThread agent INFO No
cgrouped processes running; unmounting any cgroup roots
[31/Jul/2013 18:16:32 +0000] 19950 MainThread agent INFO No
processes are being managed; Supervisor will shutdown.
[31/Jul/2013 18:16:32 +0000] 19950 MainThread agent INFO
Shutting down supervisord, pid 19969
[31/Jul/2013 18:16:32 +0000] 19950 MainThread agent INFO
Successfully shutdown supervisord, pid 19969
[31/Jul/2013 18:16:32 +0000] 19950 MainThread _cplogging INFO
[31/Jul/2013:18:16:32] ENGINE Bus STOPPING
[31/Jul/2013 18:16:32 +0000] 19950 MainThread _cplogging INFO
[31/Jul/2013:18:16:32] ENGINE HTTP Server
cherrypy._cpwsgi_server.CPWSGIServer(('ip-10-77-74-189.ec2.internal',
9000)) shut down
[31/Jul/2013 18:16:32 +0000] 19950 MainThread _cplogging INFO
[31/Jul/2013:18:16:32] ENGINE Stopped thread '_TimeoutMonitor'.
[31/Jul/2013 18:16:32 +0000] 19950 MainThread _cplogging INFO
[31/Jul/2013:18:16:32] ENGINE Bus STOPPED
[31/Jul/2013 18:16:32 +0000] 19950 MainThread _cplogging INFO
[31/Jul/2013:18:16:32] ENGINE Bus STOPPING
[31/Jul/2013 18:16:32 +0000] 19950 MainThread _cplogging INFO
[31/Jul/2013:18:16:32] ENGINE HTTP Server
cherrypy._cpwsgi_server.CPWSGIServer(('ip-10-77-74-189.ec2.internal',
9000)) already shut down
[31/Jul/2013 18:16:32 +0000] 19950 MainThread _cplogging INFO
[31/Jul/2013:18:16:32] ENGINE No thread running for None.
[31/Jul/2013 18:16:32 +0000] 19950 MainThread _cplogging INFO
[31/Jul/2013:18:16:32] ENGINE Bus STOPPED
[31/Jul/2013 18:16:32 +0000] 19950 MainThread _cplogging INFO
[31/Jul/2013:18:16:32] ENGINE Bus EXITING
[31/Jul/2013 18:16:32 +0000] 19950 MainThread _cplogging INFO
[31/Jul/2013:18:16:32] ENGINE Bus EXITED
[31/Jul/2013 18:16:32 +0000] 19950 MainThread agent INFO Agent
exiting; caught signal 15
[31/Jul/2013 18:16:32 +0000] 23193 MainThread agent INFO No
command line vars
[31/Jul/2013 18:16:32 +0000] 23193 MainThread agent INFO Missing
database jar: /usr/share/java/mysql-connector-java.jar (normal, if you're
not using this database type)
[31/Jul/2013 18:16:32 +0000] 23193 MainThread agent INFO Missing
database jar: /usr/share/java/oracle-connector-java.jar (normal, if you're
not using this database type)
[31/Jul/2013 18:16:32 +0000] 23193 MainThread agent INFO Found
database jar: /usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar
[31/Jul/2013 18:16:32 +0000] 23193 MainThread agent INFO Agent
starting as pid 23193 user root(0) group root(0).
[31/Jul/2013 18:16:32 +0000] 23193 MainThread agent INFO
Re-using pre-existing directory: /run/cloudera-scm-agent
[31/Jul/2013 18:16:32 +0000] 23193 MainThread agent INFO Agent
is running on a public cloud. Public hostname:
ec2-54-242-227-124.compute-1.amazonaws.com
[31/Jul/2013 18:16:32 +0000] 23193 MainThread agent INFO
Re-using pre-existing directory: /run/cloudera-scm-agent/cgroups
[31/Jul/2013 18:16:33 +0000] 23193 MainThread cgroups INFO Reusing
/run/cloudera-scm-agent/cgroups/memory
[31/Jul/2013 18:16:33 +0000] 23193 MainThread cgroups INFO Reusing
/run/cloudera-scm-agent/cgroups/cpu
[31/Jul/2013 18:16:33 +0000] 23193 MainThread cgroups INFO Reusing
/run/cloudera-scm-agent/cgroups/cpuacct
[31/Jul/2013 18:16:33 +0000] 23193 MainThread cgroups INFO Reusing
/run/cloudera-scm-agent/cgroups/blkio
[31/Jul/2013 18:16:33 +0000] 23193 MainThread agent INFO Found
cgroups capabilities: {'has_memory': True, 'default_memory_limit_in_bytes':
9223372036854775807, 'default_memory_soft_limit_in_bytes':
9223372036854775807, 'writable_cgroup_dot_procs': True,
'default_cpu_rt_runtime_us': 950000, 'has_cpu': True,
'default_blkio_weight': 1000, 'default_cpu_shares': 1024, 'has_cpuacct':
True, 'has_blkio': True}
[31/Jul/2013 18:16:33 +0000] 23193 MainThread agent INFO Setting
up supervisord event monitor.
[31/Jul/2013 18:16:33 +0000] 23193 MainThread filesystem_map INFO
Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs']
[31/Jul/2013 18:16:33 +0000] 23193 MainThread filesystem_map INFO Using
timeout of 2.000000
[31/Jul/2013 18:16:33 +0000] 23193 MainThread filesystem_map INFO Using
join timeout of 0.100000
[31/Jul/2013 18:16:33 +0000] 23193 MainThread filesystem_map INFO Using
tolerance of 60.000000
[31/Jul/2013 18:16:33 +0000] 23193 MainThread agent INFO Using
metrics_url_timeout_seconds of 30.000000
[31/Jul/2013 18:16:33 +0000] 23193 MainThread agent INFO Using
task_metrics_timeout_seconds of 5.000000
[31/Jul/2013 18:16:33 +0000] 23193 MainThread agent INFO Using
max_collection_wait_seconds of 10.000000
[31/Jul/2013 18:16:33 +0000] 23193 MainThread metrics INFO
Importing tasktracker metric schema from file
/usr/lib/cmf/agent/src/cmf/monitor/tasktracker/schema.json
[31/Jul/2013 18:16:33 +0000] 23193 MainThread dns_names INFO Using
timeout of 2.000000
[31/Jul/2013 18:16:33 +0000] 23193 MainThread __init__ INFO
Importing metric schema from file
/usr/lib/cmf/agent/src/cmf/monitor/schema.json
[31/Jul/2013 18:16:33 +0000] 23193 MainThread agent INFO
Supervised processes will inherit the following environment: {'LANG':
'en_US.UTF-8', 'TERM': 'vt100', 'CMF_CONF_DIR': '/etc/cloudera-scm-agent',
'SHLVL': '1', 'PWD': '/', 'PATH':
'/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games'}
[31/Jul/2013 18:16:33 +0000] 23193 MainThread agent INFO
Supervised processes will add the following to their environment:
{'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_MR2_HOME':
'/usr/lib/hadoop-mapreduce', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper',
'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/usr/share/cmf',
'CDH_IMPALA_HOME': '/usr/lib/impala', 'CLOUDERA_MYSQL_CONNECTOR_JAR':
'/usr/share/java/mysql-connector-java.jar', 'CDH_YARN_HOME':
'/usr/lib/hadoop-yarn', 'CMF_PACKAGE_DIR': '/usr/lib/cmf/service',
'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'CDH_SOLR_HOME': '/usr/lib/solr',
'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2',
'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE
HDFS_DFS_DIR_NOT_EMPTY JOBTRACKER_IN_STANDBY_MODE', 'CDH_MR1_HOME':
'/usr/lib/hadoop-0.20-mapreduce', 'CLOUDERA_ORACLE_CONNECTOR_JAR':
'/usr/share/java/oracle-connector-java.jar', 'CDH_HUE_HOME':
'/usr/share/hue', 'CDH_HIVE_HOME': '/usr/lib/hive', 'CDH_HTTPFS_HOME':
'/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop',
'JSVC_HOME': '/usr/libexec/bigtop-utils', 'HIVE_DEFAULT_XML':
'/etc/hive/conf.dist/hive-default.xml', 'CDH_FLUME_HOME':
'/usr/lib/flume-ng', 'CLOUDERA_POSTGRESQL_JDBC_JAR':
'/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'TOMCAT_HOME':
'/usr/lib/bigtop-tomcat', 'CDH_HBASE_HOME': '/usr/lib/hbase',
'CDH_HCAT_HOME': '/usr/lib/hcatalog'}
[31/Jul/2013 18:16:33 +0000] 23193 MainThread agent INFO To
override these variables, use /etc/cloudera-scm-agent/config.ini.
Environment variables for CDH locations are not used when CDH is installed
from parcels.
[31/Jul/2013 18:16:33 +0000] 23193 MainThread agent INFO
Re-using pre-existing directory: /run/cloudera-scm-agent/process
[31/Jul/2013 18:16:33 +0000] 23193 MainThread agent INFO
Re-using pre-existing directory: /run/cloudera-scm-agent/supervisor
[31/Jul/2013 18:16:33 +0000] 23193 MainThread agent INFO
Re-using pre-existing directory: /run/cloudera-scm-agent/supervisor/include
[31/Jul/2013 18:16:33 +0000] 23193 MainThread agent INFO Failed
to connect to previous supervisor.
[31/Jul/2013 18:16:35 +0000] 23193 MainThread agent INFO Trying
to connect to newly launched supervisor (Attempt 1)
[31/Jul/2013 18:16:35 +0000] 23193 MainThread agent INFO
Successfully connected to supervisor
[31/Jul/2013 18:16:35 +0000] 23193 MainThread _cplogging INFO
[31/Jul/2013:18:16:35] ENGINE Bus STARTING
[31/Jul/2013 18:16:35 +0000] 23193 MainThread _cplogging INFO
[31/Jul/2013:18:16:35] ENGINE Started monitor thread '_TimeoutMonitor'.
[31/Jul/2013 18:16:35 +0000] 23193 MainThread _cplogging INFO
[31/Jul/2013:18:16:35] ENGINE Serving on
ec2-54-242-227-124.compute-1.amazonaws.com:9000
[31/Jul/2013 18:16:35 +0000] 23193 MainThread _cplogging INFO
[31/Jul/2013:18:16:35] ENGINE Bus STARTED
[31/Jul/2013 18:16:35 +0000] 23193 MainThread __init__ INFO New
monitor: (<cmf.monitor.host.hostmonitor object@0x3230f50>,)
[31/Jul/2013 18:16:35 +0000] 23193 MainThread agent WARNING Setting
default socket timeout to 30!
[31/Jul/2013 18:16:35 +0000] 23193 MonitorDaemon-Scheduler __init__
INFO Monitor ready to report: (<cmf.monitor.host.HostMonitor object at
0x3230f50>,)
[31/Jul/2013 18:16:35 +0000] 23193 MainThread parcel INFO Agent
creates users/groups and applies file permissions: True
[31/Jul/2013 18:16:35 +0000] 23193 MainThread downloader INFO
Downloader path: /opt/cloudera/parcel-cache
[31/Jul/2013 18:16:35 +0000] 23193 MainThread parcel_cache INFO Using
/opt/cloudera/parcel-cache for parcel cache
[31/Jul/2013 18:16:40 +0000] 23193 Monitor-HostMonitor network_interfaces
INFO NIC iface eth0 doesn't support ETHTOOL (95)
Any help would be greatly appreciated!
Thanks,
Ryan
