Grokbase › Groups › Hadoop › scm-users › June 2013
FAQ
I just upgraded the CM server in EC2 to 4.6 from 4.5.2. I am trying to
upgrade the node agents to the 4.6 using the wizard. I have 11 EC2 nodes in
the cluster. When it gets to the Upgrade Hosts page, it cannot connect to
any of the nodes in EC2. The names of all the nodes have been replaced with
names like "i-4f792e33". I can see the original hosts with their correct
names on the Hosts page. What can I do to fix this?

Thanks,
Ben

Search Discussions

  • Benjamin Kim at Jun 8, 2013 at 12:51 am ⇧
    I was able to fix the problem by going into the scm db and modifying the
    hosts table to have the identifier also be the name. But now, I'm having
    another issue. The CM Agents will not start up.

    Here is the log trace returned.

    /tmp/scm_prepare_node.jChJxhLv
    using SSH_CLIENT to get the SCM hostname: 10.124.243.21 37046 22
    opening logging file descriptor
    Starting installation script...Acquiring installation lock...
    *BEGIN flock 4 *
    *END (0) *
    Detecting root privileges...
    effective UID is 0
    Detecting distribution...
    *BEGIN grep Tikanga /etc/redhat-release *
    *END (1) *
    *BEGIN grep 'CentOS release 5' /etc/redhat-release *
    *END (1) *
    *BEGIN grep 'Scientific Linux release 5' /etc/redhat-release *
    *END (1) *
    *BEGIN grep Santiago /etc/redhat-release *
    *END (1) *
    *BEGIN grep 'CentOS Linux release 6' /etc/redhat-release *
    *END (1) *
    *BEGIN grep 'CentOS release 6' /etc/redhat-release *
    CentOS release 6.3 (Final)
    *END (0) *
    /etc/redhat-release ==> CentOS 6
    Detecting Cloudera Manager Server...
    *BEGIN host -t PTR 10.124.243.21 *
    21.243.124.10.in-addr.arpa domain name pointer
    ip-10-124-243-21.ec2.internal.
    *END (0) *
    using ip-10-124-243-21.ec2.internal as scm server hostname
    *BEGIN which python *
    /usr/bin/python
    *END (0) *
    *BEGIN python -c 'import socket; import sys; s =
    socket.socket(socket.AF_INET); s.settimeout(5.0); s.connect((sys.argv[1],
    int(sys.argv[2]))); s.close();' ip-10-124-243-21.ec2.internal 7182 *
    *END (0) *
    *BEGIN which wget *
    /usr/bin/wget
    *END (0) *
    *BEGIN wget -qO- -T 1 -t 1
    http://169.254.169.254/latest/meta-data/public-hostname && /bin/echo *
    ec2-107-21-133-38.compute-1.amazonaws.com
    *END (0) *
    *BEGIN wget -qO- -T 1 -t 1
    http://169.254.169.254/latest/meta-data/instance-id && /bin/echo *
    i-05653279
    *END (0) *
    Performing cloud-specific configuration...
    *BEGIN /sbin/chkconfig iptables *
    *END (1) *
    *BEGIN /sbin/service iptables status | grep Chain *
    Chain INPUT (policy ACCEPT)
    Chain FORWARD (policy ACCEPT)
    Chain OUTPUT (policy ACCEPT)
    Chain RH-Firewall-1-INPUT (2 references)
    *END (0) *
    *BEGIN /sbin/service iptables stop *
    iptables: Flushing firewall rules: [ OK ]
    iptables: Setting chains to policy ACCEPT: filter [ OK ]
    iptables: Unloading modules: [ OK ]
    *END (0) *
    *BEGIN getenforce *
    Disabled
    *END (0) *
    *BEGIN grep -e ^SELINUX=enforcing /etc/selinux/config *
    *END (1) *
    *BEGIN df -h | grep -o -E "^/dev/[^[:space:]]*" *
    /dev/xvde1
    /dev/md0
    *END (0) *
    *BEGIN awk 'FNR > 2 {print $NF}' /proc/partitions *
    xvdf
    xvdf1
    xvdf2
    xvdg
    xvdg1
    xvdg2
    xvde
    xvde1
    md0
    *END (0) *
    Installing package repositories...
    validating format of repository file
    /tmp/scm_prepare_node.jChJxhLv/repos/rhel6/cloudera-manager.repo
    installing repository file
    /tmp/scm_prepare_node.jChJxhLv/repos/rhel6/cloudera-manager.repo
    repository file
    /tmp/scm_prepare_node.jChJxhLv/repos/rhel6/cloudera-manager.repo installed
    Refreshing package metadata...
    *BEGIN yum clean all *
    Loaded plugins: fastestmirror, presto
    Cleaning repos: base cloudera-cdh4 cloudera-impala cloudera-manager epel
    extras
    : updates
    Cleaning up Everything
    0 delta-package files removed, by presto
    *END (0) *
    *BEGIN rm -Rf /var/cache/yum/x86_64 *
    *END (0) *
    *BEGIN yum makecache *
    Loaded plugins: fastestmirror, presto
    Metadata Cache Created
    *END (0) *
    Installing jdk package...
    *BEGIN yum list installed jdk *
    Loaded plugins: fastestmirror, presto
    Installed Packages
    jdk.x86_64 2000:1.6.0_31-fcs @cloudera-manager
    *END (0) *
    *BEGIN echo jdk cloudera-manager-agent cloudera-manager-daemons | grep jdk *
    jdk cloudera-manager-agent cloudera-manager-daemons
    *END (0) *
    *BEGIN yum info jdk *
    Loaded plugins: fastestmirror, presto
    Determining fastest mirrors
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.fedoraproject.org
    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Installed Packages
    Name : jdk
    Arch : x86_64
    Epoch : 2000
    Version : 1.6.0_31
    Release : fcs
    Size : 143 M
    Repo : installed
    From repo : cloudera-manager
    Summary : Java(TM) Platform Standard Edition Development Kit
    URL : http://java.sun.com/
    License : Copyright (c) 2011, Oracle and/or its affiliates. All rights
    : reserved. Also under other license(s) as shown at the Description
    : field.
    Description : The Java Platform Standard Edition Development Kit (JDK)
    includes
    : both the runtime environment (Java virtual machine, the Java
    : platform classes and supporting files) and development tools
    : (compilers, debuggers, tool libraries and other tools).
    :
    : The JDK is a development environment for building applications,
    : applets and components that can be deployed with the Java Platform
    : Standard Edition Runtime Environment.

    *END (0) *
    *BEGIN yum -y install jdk.x86_64 *
    Loaded plugins: fastestmirror, presto
    Loading mirror speeds from cached hostfile
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.fedoraproject.org
    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Setting up Install Process
    Package 2000:jdk-1.6.0_31-fcs.x86_64 already installed and latest version
    Nothing to do
    *END (0) *
    remote package jdk installed
    Installing cloudera-manager-agent package...
    *BEGIN yum list installed cloudera-manager-agent *
    Loaded plugins: fastestmirror, presto
    Installed Packages
    cloudera-manager-agent.x86_64 4.6.0-1.cm460.p0.140 @cloudera-manager
    *END (0) *
    *BEGIN echo jdk cloudera-manager-agent cloudera-manager-daemons | grep
    cloudera-manager-agent *
    *END (0) *
    jdk cloudera-manager-agent cloudera-manager-daemons
    *BEGIN yum info cloudera-manager-agent *
    Loaded plugins: fastestmirror, presto
    Loading mirror speeds from cached hostfile
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.fedoraproject.org
    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Installed Packages
    Name : cloudera-manager-agent
    Arch : x86_64
    Version : 4.6.0
    Release : 1.cm460.p0.140
    Size : 25 M
    Repo : installed
    From repo : cloudera-manager
    Summary : The Cloudera Manager Agent
    URL : http://www.cloudera.com
    License : Proprietary
    Description : The Cloudera Manager Agent.
    :
    : The Agent is deployed to machines running services managed by
    : Cloudera Manager.

    *END (0) *
    Version : 4.6.0
    Release : 1.cm460.p0.140
    *BEGIN yum -y install cloudera-manager-agent *
    Loaded plugins: fastestmirror, presto
    Loading mirror speeds from cached hostfile
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.fedoraproject.org
    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Setting up Install Process
    Package cloudera-manager-agent-4.6.0-1.cm460.p0.140.x86_64 already
    installed and latest version
    Nothing to do
    *END (0) *
    remote package cloudera-manager-agent installed
    Installing cloudera-manager-daemons package...
    *BEGIN yum list installed cloudera-manager-daemons *
    Loaded plugins: fastestmirror, presto
    Installed Packages
    cloudera-manager-daemons.x86_64 4.6.0-1.cm460.p0.140 @cloudera-manager
    *END (0) *
    *BEGIN echo jdk cloudera-manager-agent cloudera-manager-daemons | grep
    cloudera-manager-daemons *
    jdk cloudera-manager-agent cloudera-manager-daemons
    *END (0) *
    *BEGIN yum info cloudera-manager-daemons *
    Loaded plugins: fastestmirror, presto
    Loading mirror speeds from cached hostfile
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.fedoraproject.org
    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Installed Packages
    Name : cloudera-manager-daemons
    Arch : x86_64
    Version : 4.6.0
    Release : 1.cm460.p0.140
    Size : 175 M
    Repo : installed
    From repo : cloudera-manager
    Summary : Provides daemons for monitoring Hadoop and related tools.
    URL : http://www.cloudera.com
    License : Proprietary
    Description : This package includes daemons for monitoring and managing
    Hadoop.

    *END (0) *
    Version : 4.6.0
    Release : 1.cm460.p0.140
    *BEGIN yum -y install cloudera-manager-daemons *
    Loaded plugins: fastestmirror, presto
    Loading mirror speeds from cached hostfile
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.fedoraproject.org
    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Setting up Install Process
    Package cloudera-manager-daemons-4.6.0-1.cm460.p0.140.x86_64 already
    installed and latest version
    Nothing to do
    *END (0) *
    remote package cloudera-manager-daemons installed
    Configuring Cloudera Manager Agent...
    *BEGIN sed -e 's/\(CMF_AGENT_ARGS=\).*/\1"--host_id i-05653279"/' -i
    /etc/default/cloudera-scm-agent *
    *END (0) *
    *BEGIN grep server_host=ip-10-124-243-21.ec2.internal
    /etc/cloudera-scm-agent/config.ini *
    server_host=ip-10-124-243-21.ec2.internal
    *END (0) *
    scm agent is already configured
    Starting Cloudera Manager Agent...
    *BEGIN /sbin/service cloudera-scm-agent status | grep running *
    *END (1) *
    *BEGIN /sbin/service cloudera-scm-agent start *
    Starting cloudera-scm-agent: [FAILED]
    *END (1) *
    agent logs:
    *BEGIN tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.out | sed
    's/^/>>/' *
    /usr/lib64/cmf/agent/src/cmf/agent.py:31: DeprecationWarning: the sha
    module is deprecated; use the hashlib module instead
    import sha
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO SCM Agent
    Version: 4.6.0
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Using directory:
    /var/run/cloudera-scm-agent
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Using supervisor
    binary path: /usr/lib64/cmf/agent/src/cmf/../../build/env/bin/supervisord
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Adding env vars
    that start with CMF_AGENT_
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: HIVE_DEFAULT_XML:
    /etc/hive/conf.dist/hive-default.xml
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: MGMT_HOME: /usr/share/cmf
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HTTPFS_HOME: /usr/lib/hadoop-httpfs
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_FLUME_HOME: /usr/lib/flume-ng
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HBASE_HOME: /usr/lib/hbase
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HUE_HOME: /usr/share/hue
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: TOMCAT_HOME: /usr/lib/bigtop-tomcat
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_PIG_HOME: /usr/lib/pig
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HIVE_HOME: /usr/lib/hive
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_YARN_HOME: /usr/lib/hadoop-yarn
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HADOOP_BIN: /usr/bin/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_OOZIE_HOME: /usr/lib/oozie
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_ZOOKEEPER_HOME: /usr/lib/zookeeper
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HUE_PLUGINS_HOME: /usr/lib/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: JSVC_HOME: /usr/libexec/bigtop-utils
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HDFS_HOME: /usr/lib/hadoop-hdfs
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_MR1_HOME: /usr/lib/hadoop-0.20-mapreduce
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_MR2_HOME: /usr/lib/hadoop-mapreduce
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HADOOP_HOME: /usr/lib/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_IMPALA_HOME: /usr/lib/impala
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Logging to
    /var/log/cloudera-scm-agent/cloudera-scm-agent.log
    Traceback (most recent call last):
    File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 2391, in <module>
    main()
    File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 2375, in main
    agent.start()
    File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 329, in start
    self.add_db_driver_jars()
    File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 1400, in
    add_db_driver_ja>>/usr/lib64/cmf/agent/src/cmf/agent.py:31:
    DeprecationWarning: the sha module is deprecated; use the hashlib module
    instead
    import sha
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO SCM Agent
    Version: 4.6.0
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Using directory:
    /var/run/cloudera-scm-agent
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Using supervisor
    binary path: /usr/lib64/cmf/agent/src/cmf/../../build/env/bin/supervisord
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Adding env vars
    that start with CMF_AGENT_
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: HIVE_DEFAULT_XML:
    /etc/hive/conf.dist/hive-default.xml
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: MGMT_HOME: /usr/share/cmf
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HTTPFS_HOME: /usr/lib/hadoop-httpfs
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_FLUME_HOME: /usr/lib/flume-ng
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HBASE_HOME: /usr/lib/hbase
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HUE_HOME: /usr/share/hue
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: TOMCAT_HOME: /usr/lib/bigtop-tomcat
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_PIG_HOME: /usr/lib/pig
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HIVE_HOME: /usr/lib/hive
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_YARN_HOME: /usr/lib/hadoop-yarn
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HADOOP_BIN: /usr/bin/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_OOZIE_HOME: /usr/lib/oozie
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_ZOOKEEPER_HOME: /usr/lib/zookeeper
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HUE_PLUGINS_HOME: /usr/lib/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: JSVC_HOME: /usr/libexec/bigtop-utils
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HDFS_HOME: /usr/lib/hadoop-hdfs
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_MR1_HOME: /usr/lib/hadoop-0.20-mapreduce
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_MR2_HOME: /usr/lib/hadoop-mapreduce
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HADOOP_HOME: /usr/lib/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_IMPALA_HOME: /usr/lib/impala
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Logging to
    /var/log/cloudera-scm-agent/cloudera-scm-agent.log
    Traceback (most recent call last):
    File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 2391, in <module>
    main()
    File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 2375, in main
    agent.start()
    File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 329, in start
    self.add_db_driver_jars()
    File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 1400, in
    add_db_driver_jars
    for jar in (mysql_jar, oracle_jar, postgresql_jar):
    UnboundLocalError: local variable 'postgresql_jar' referenced before
    assignment
    rs
    for jar in (mysql_jar, oracle_jar, postgresql_jar):
    UnboundLocalError: local variable 'postgresql_jar' referenced before
    assignment
    *END (0) *
    *BEGIN tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.log | sed
    's/^/>>/' *
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Deleting process
    1014-mapreduce-TASKTRACKER
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Triggering
    supervisord update.
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Retiring process
    1014-mapreduce-TASKTRACKER
    [07/Jun/2013 22:59:32 +0000] 695 MonitorDaemon-Reporter throttling_logger
    ERROR Error sending messages to firehose:
    mgmt1-HOSTMONITOR-d889e476015d035afbfed2002d76de2d
    Traceback (most recent call last):
    File "/usr/lib64/cmf/agent/src/cmf/monitor/firehose.py", line 67, in
    send
    self._requestor.request('sendAgentMessages', dict(messages=messages))
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py",
    line 145, in request
    return self.issue_request(call_request, message_name, request_datum)
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py",
    line 256, in issue_request
    call_response = self.transceiver.transceive(call_request)
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py",
    line 485, in transceive
    result = self.read_framed_message()
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py",
    line 489, in read_framed_message
    response = self.conn.getresponse()
    File "/usr/lib64/python2.6/httplib.py", line 990, in getresponse
    response.begin()
    File "/usr/lib64/python2.6/httplib.py", line 391, in begin
    version, status, reason = self._read_status()
    File "/usr/lib64/python2.6/httplib.py", line 355, in _read_status
    raise BadStatusLine(line)
    BadStatusLine
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Stopping agent...
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO No cgrouped
    processes running; unmounting any cgroup roots
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO No processes are
    being managed; Supervisor will shutdown.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Shutting down
    supervisord, pid 715
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Successfully
    shutdown supervisord, pid 715
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE HTTP Server
    cherrypy._cpwsgi_server.CPWSGIServer(('dn6.lab10.adconion.com', 9000)) shut
    down
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Stopped thread '_TimeoutMonitor'.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE HTTP Server
    cherrypy._cpwsgi_server.CPWSGIServer(('dn6.lab10.adconion.com', 9000))
    already shut down
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE No thread running for None.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus EXITING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus EXITED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Agent exiting;
    caught signal 15
    [07/Jun/2013 23:02:30 +0000] 12620 MainThread agent INFO No command line
    vars
    [07/Jun/2013 23:02:30 +0000] 12620 Dummy-1 agent INFO Stopping agent...
    [07/Jun/2013 23:20:25 +0000] 14136 MainThread agent INFO No command line
    vars
    [07/Jun/2013 23:20:25 +0000] 14136 Dummy-1 agent INFO Stopping agent...
    [07/Jun/2013 23:51:20 +0000] 15580 MainThread agent INFO No command line
    vars
    [07/Jun/2013 23:51:20 +0000] 15580 Dummy-1 agent INFO Stopping agent...
    [07/Jun/2013 23:53:21 +0000] 16382 MainThread agent INFO No command line
    vars
    [07/Jun/2013 23:53:21 +0000] 16382 Dummy-1 agent INFO Stopping agent...
    [08/Jun/2013 00:05:13 +0000] 17261 MainThread agent INFO No command line
    vars
    [08/Jun/2013 00:05:13 +0000] 17261 Dummy-1 agent INFO Stopping agent...
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO No command line
    vars
    [08/Jun/2013 00:43:29 +0000] 20804 Dummy-1 agent INFO Stopping agent...
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Deleting process
    1014-mapreduce-TASKTRACKER
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Triggering
    supervisord update.
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Retiring process
    1014-mapreduce-TASKTRACKER
    [07/Jun/2013 22:59:32 +0000] 695 MonitorDaemon-Reporter throttling_logger
    ERROR Error sending messages to firehose:
    mgmt1-HOSTMONITOR-d889e476015d035afbfed2002d76de2d
    Traceback (most recent call last):
    File "/usr/lib64/cmf/agent/src/cmf/monitor/firehose.py", line 67, in
    send
    self._requestor.request('sendAgentMessages', dict(messages=messages))
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py",
    line 145, in request
    return self.issue_request(call_request, message_name, request_datum)
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py",
    line 256, in issue_request
    call_response = self.transceiver.transceive(call_request)
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py",
    line 485, in transceive
    result = self.read_framed_message()
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py",
    line 489, in read_framed_message
    response = self.conn.getresponse()
    File "/usr/lib64/python2.6/httplib.py", line 990, in getresponse
    response.begin()
    File "/usr/lib64/python2.6/httplib.py", line 391, in begin
    version, status, reason = self._read_status()
    File "/usr/lib64/python2.6/httplib.py", line 355, in _read_status
    raise BadStatusLine(line)
    BadStatusLine
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Stopping agent...
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO No cgrouped
    processes running; unmounting any cgroup roots
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO No processes are
    being managed; Supervisor will shutdown.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Shutting down
    supervisord, pid 715
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Successfully
    shutdown supervisord, pid 715
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE HTTP Server
    cherrypy._cpwsgi_server.CPWSGIServer(('dn6.lab10.adconion.com', 9000)) shut
    down
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Stopped thread '_TimeoutMonitor'.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE HTTP Server
    cherrypy._cpwsgi_server.CPWSGIServer(('dn6.lab10.adconion.com', 9000))
    already shut down
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE No thread running for None.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus EXITING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus EXITED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Agent exiting;
    caught signal 15
    [07/Jun/2013 23:02:30 +0000] 12620 MainThread agent INFO No command line
    vars
    [07/Jun/2013 23:02:30 +0000] 12620 Dummy-1 agent INFO Stopping agent...
    [07/Jun/2013 23:20:25 +0000] 14136 MainThread agent INFO No command line
    vars
    [07/Jun/2013 23:20:25 +0000] 14136 Dummy-1 agent INFO Stopping agent...
    [07/Jun/2013 23:51:20 +0000] 15580 MainThread agent INFO No command line
    vars
    [07/Jun/2013 23:51:20 +0000] 15580 Dummy-1 agent INFO Stopping agent...
    [07/Jun/2013 23:53:21 +0000] 16382 MainThread agent INFO No command line
    vars
    [07/Jun/2013 23:53:21 +0000] 16382 Dummy-1 agent INFO Stopping agent...
    [08/Jun/2013 00:05:13 +0000] 17261 MainThread agent INFO No command line
    vars
    [08/Jun/2013 00:05:13 +0000] 17261 Dummy-1 agent INFO Stopping agent...
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO No command line
    vars
    [08/Jun/2013 00:43:29 +0000] 20804 Dummy-1 agent INFO Stopping agent...
    *END (0) *
    end of agent logs.
    scm agent could not be started, giving up
    waiting for rollback request

    I hope you can help.

    Thanks,
    Ben
    On Friday, June 7, 2013 3:55:09 PM UTC-7, Benjamin Kim wrote:

    I just upgraded the CM server in EC2 to 4.6 from 4.5.2. I am trying to
    upgrade the node agents to the 4.6 using the wizard. I have 11 EC2 nodes in
    the cluster. When it gets to the Upgrade Hosts page, it cannot connect to
    any of the nodes in EC2. The names of all the nodes have been replaced with
    names like "i-4f792e33". I can see the original hosts with their correct
    names on the Hosts page. What can I do to fix this?

    Thanks,
    Ben
  • Darren Lo at Jun 10, 2013 at 5:50 am ⇧
    Hi Ben,

    Sorry for the toruble, and thanks for finding this bug! I've reproduced the
    error locally and I will make sure this is fixed in 4.6.1.

    In the meantime, you can get past this bug by removing the export line for
    CLOUDERA_POSTGRESQL_JDBC_JAR from /etc/default/cloudera-scm-agent on all
    hosts. It isn't necessary anymore in 4.6. The agent code will correctly
    locate the jar in the cm lib dir by default (normally /usr/share/cmf/lib).

    Let me know if it works!

    Thanks,
    Darren

    On Fri, Jun 7, 2013 at 5:51 PM, Benjamin Kim wrote:

    I was able to fix the problem by going into the scm db and modifying the
    hosts table to have the identifier also be the name. But now, I'm having
    another issue. The CM Agents will not start up.

    Here is the log trace returned.

    /tmp/scm_prepare_node.jChJxhLv
    using SSH_CLIENT to get the SCM hostname: 10.124.243.21 37046 22
    opening logging file descriptor
    Starting installation script...Acquiring installation lock...
    *BEGIN flock 4 *
    *END (0) *
    Detecting root privileges...
    effective UID is 0
    Detecting distribution...
    *BEGIN grep Tikanga /etc/redhat-release *
    *END (1) *
    *BEGIN grep 'CentOS release 5' /etc/redhat-release *
    *END (1) *
    *BEGIN grep 'Scientific Linux release 5' /etc/redhat-release *
    *END (1) *
    *BEGIN grep Santiago /etc/redhat-release *
    *END (1) *
    *BEGIN grep 'CentOS Linux release 6' /etc/redhat-release *
    *END (1) *
    *BEGIN grep 'CentOS release 6' /etc/redhat-release *
    CentOS release 6.3 (Final)
    *END (0) *
    /etc/redhat-release ==> CentOS 6
    Detecting Cloudera Manager Server...
    *BEGIN host -t PTR 10.124.243.21 *
    21.243.124.10.in-addr.arpa domain name pointer
    ip-10-124-243-21.ec2.internal.
    *END (0) *
    using ip-10-124-243-21.ec2.internal as scm server hostname
    *BEGIN which python *
    /usr/bin/python
    *END (0) *
    *BEGIN python -c 'import socket; import sys; s =
    socket.socket(socket.AF_INET); s.settimeout(5.0); s.connect((sys.argv[1],
    int(sys.argv[2]))); s.close();' ip-10-124-243-21.ec2.internal 7182 *
    *END (0) *
    *BEGIN which wget *
    /usr/bin/wget
    *END (0) *
    *BEGIN wget -qO- -T 1 -t 1
    http://169.254.169.254/latest/meta-data/public-hostname && /bin/echo *
    ec2-107-21-133-38.compute-1.amazonaws.com
    *END (0) *
    *BEGIN wget -qO- -T 1 -t 1
    http://169.254.169.254/latest/meta-data/instance-id && /bin/echo *
    i-05653279
    *END (0) *
    Performing cloud-specific configuration...
    *BEGIN /sbin/chkconfig iptables *
    *END (1) *
    *BEGIN /sbin/service iptables status | grep Chain *
    Chain INPUT (policy ACCEPT)
    Chain FORWARD (policy ACCEPT)
    Chain OUTPUT (policy ACCEPT)
    Chain RH-Firewall-1-INPUT (2 references)
    *END (0) *
    *BEGIN /sbin/service iptables stop *
    iptables: Flushing firewall rules: [60G[ [0;32m OK [0;39m]
    iptables: Setting chains to policy ACCEPT: filter [60G[ [0;32m OK [0;39m]
    iptables: Unloading modules: [60G[ [0;32m OK [0;39m]
    *END (0) *
    *BEGIN getenforce *
    Disabled
    *END (0) *
    *BEGIN grep -e ^SELINUX=enforcing /etc/selinux/config *
    *END (1) *
    *BEGIN df -h | grep -o -E "^/dev/[^[:space:]]*" *
    /dev/xvde1
    /dev/md0
    *END (0) *
    *BEGIN awk 'FNR > 2 {print $NF}' /proc/partitions *
    xvdf
    xvdf1
    xvdf2
    xvdg
    xvdg1
    xvdg2
    xvde
    xvde1
    md0
    *END (0) *
    Installing package repositories...
    validating format of repository file
    /tmp/scm_prepare_node.jChJxhLv/repos/rhel6/cloudera-manager.repo
    installing repository file
    /tmp/scm_prepare_node.jChJxhLv/repos/rhel6/cloudera-manager.repo
    repository file
    /tmp/scm_prepare_node.jChJxhLv/repos/rhel6/cloudera-manager.repo installed
    Refreshing package metadata...
    *BEGIN yum clean all *
    Loaded plugins: fastestmirror, presto
    Cleaning repos: base cloudera-cdh4 cloudera-impala cloudera-manager epel
    extras
    : updates
    Cleaning up Everything
    0 delta-package files removed, by presto
    *END (0) *
    *BEGIN rm -Rf /var/cache/yum/x86_64 *
    *END (0) *
    *BEGIN yum makecache *
    Loaded plugins: fastestmirror, presto
    Metadata Cache Created
    *END (0) *
    Installing jdk package...
    *BEGIN yum list installed jdk *
    Loaded plugins: fastestmirror, presto
    Installed Packages
    jdk.x86_64 2000:1.6.0_31-fcs @cloudera-manager
    *END (0) *
    *BEGIN echo jdk cloudera-manager-agent cloudera-manager-daemons | grep
    jdk *
    jdk cloudera-manager-agent cloudera-manager-daemons
    *END (0) *
    *BEGIN yum info jdk *
    Loaded plugins: fastestmirror, presto
    Determining fastest mirrors
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.fedoraproject.org
    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Installed Packages
    Name : jdk
    Arch : x86_64
    Epoch : 2000
    Version : 1.6.0_31
    Release : fcs
    Size : 143 M
    Repo : installed
    From repo : cloudera-manager
    Summary : Java(TM) Platform Standard Edition Development Kit
    URL : http://java.sun.com/
    License : Copyright (c) 2011, Oracle and/or its affiliates. All rights
    : reserved. Also under other license(s) as shown at the Description
    : field.
    Description : The Java Platform Standard Edition Development Kit (JDK)
    includes
    : both the runtime environment (Java virtual machine, the Java
    : platform classes and supporting files) and development tools
    : (compilers, debuggers, tool libraries and other tools).
    :
    : The JDK is a development environment for building applications,
    : applets and components that can be deployed with the Java Platform
    : Standard Edition Runtime Environment.

    *END (0) *
    *BEGIN yum -y install jdk.x86_64 *
    Loaded plugins: fastestmirror, presto
    Loading mirror speeds from cached hostfile
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.fedoraproject.org
    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Setting up Install Process
    Package 2000:jdk-1.6.0_31-fcs.x86_64 already installed and latest version
    Nothing to do
    *END (0) *
    remote package jdk installed
    Installing cloudera-manager-agent package...
    *BEGIN yum list installed cloudera-manager-agent *
    Loaded plugins: fastestmirror, presto
    Installed Packages
    cloudera-manager-agent.x86_64 4.6.0-1.cm460.p0.140 @cloudera-manager
    *END (0) *
    *BEGIN echo jdk cloudera-manager-agent cloudera-manager-daemons | grep
    cloudera-manager-agent *
    *END (0) *
    jdk cloudera-manager-agent cloudera-manager-daemons
    *BEGIN yum info cloudera-manager-agent *
    Loaded plugins: fastestmirror, presto
    Loading mirror speeds from cached hostfile
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.fedoraproject.org
    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Installed Packages
    Name : cloudera-manager-agent
    Arch : x86_64
    Version : 4.6.0
    Release : 1.cm460.p0.140
    Size : 25 M
    Repo : installed
    From repo : cloudera-manager
    Summary : The Cloudera Manager Agent
    URL : http://www.cloudera.com
    License : Proprietary
    Description : The Cloudera Manager Agent.
    :
    : The Agent is deployed to machines running services managed by
    : Cloudera Manager.

    *END (0) *
    Version : 4.6.0
    Release : 1.cm460.p0.140
    *BEGIN yum -y install cloudera-manager-agent *
    Loaded plugins: fastestmirror, presto
    Loading mirror speeds from cached hostfile
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.fedoraproject.org
    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Setting up Install Process
    Package cloudera-manager-agent-4.6.0-1.cm460.p0.140.x86_64 already
    installed and latest version
    Nothing to do
    *END (0) *
    remote package cloudera-manager-agent installed
    Installing cloudera-manager-daemons package...
    *BEGIN yum list installed cloudera-manager-daemons *
    Loaded plugins: fastestmirror, presto
    Installed Packages
    cloudera-manager-daemons.x86_64 4.6.0-1.cm460.p0.140 @cloudera-manager
    *END (0) *
    *BEGIN echo jdk cloudera-manager-agent cloudera-manager-daemons | grep
    cloudera-manager-daemons *
    jdk cloudera-manager-agent cloudera-manager-daemons
    *END (0) *
    *BEGIN yum info cloudera-manager-daemons *
    Loaded plugins: fastestmirror, presto
    Loading mirror speeds from cached hostfile
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.fedoraproject.org
    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Installed Packages
    Name : cloudera-manager-daemons
    Arch : x86_64
    Version : 4.6.0
    Release : 1.cm460.p0.140
    Size : 175 M
    Repo : installed
    From repo : cloudera-manager
    Summary : Provides daemons for monitoring Hadoop and related tools.
    URL : http://www.cloudera.com
    License : Proprietary
    Description : This package includes daemons for monitoring and managing
    Hadoop.

    *END (0) *
    Version : 4.6.0
    Release : 1.cm460.p0.140
    *BEGIN yum -y install cloudera-manager-daemons *
    Loaded plugins: fastestmirror, presto
    Loading mirror speeds from cached hostfile
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.fedoraproject.org
    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Setting up Install Process
    Package cloudera-manager-daemons-4.6.0-1.cm460.p0.140.x86_64 already
    installed and latest version
    Nothing to do
    *END (0) *
    remote package cloudera-manager-daemons installed
    Configuring Cloudera Manager Agent...
    *BEGIN sed -e 's/\(CMF_AGENT_ARGS=\).*/\1"--host_id i-05653279"/' -i
    /etc/default/cloudera-scm-agent *
    *END (0) *
    *BEGIN grep server_host=ip-10-124-243-21.ec2.internal
    /etc/cloudera-scm-agent/config.ini *
    server_host=ip-10-124-243-21.ec2.internal
    *END (0) *
    scm agent is already configured
    Starting Cloudera Manager Agent...
    *BEGIN /sbin/service cloudera-scm-agent status | grep running *
    *END (1) *
    *BEGIN /sbin/service cloudera-scm-agent start *
    Starting cloudera-scm-agent: [60G[ [0;31mFAILED [0;39m]
    *END (1) *
    agent logs:
    *BEGIN tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.out |
    sed 's/^/>>/' *
    /usr/lib64/cmf/agent/src/cmf/agent.py:31: DeprecationWarning: the sha
    module is deprecated; use the hashlib module instead
    import sha
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO SCM Agent
    Version: 4.6.0
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Using
    directory: /var/run/cloudera-scm-agent
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Using
    supervisor binary path:
    /usr/lib64/cmf/agent/src/cmf/../../build/env/bin/supervisord
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Adding env vars
    that start with CMF_AGENT_
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: HIVE_DEFAULT_XML:
    /etc/hive/conf.dist/hive-default.xml
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: MGMT_HOME: /usr/share/cmf
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HTTPFS_HOME: /usr/lib/hadoop-httpfs
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_FLUME_HOME: /usr/lib/flume-ng
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HBASE_HOME: /usr/lib/hbase
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HUE_HOME: /usr/share/hue
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: TOMCAT_HOME: /usr/lib/bigtop-tomcat
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_PIG_HOME: /usr/lib/pig
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HIVE_HOME: /usr/lib/hive
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_YARN_HOME: /usr/lib/hadoop-yarn
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HADOOP_BIN: /usr/bin/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_OOZIE_HOME: /usr/lib/oozie
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_ZOOKEEPER_HOME: /usr/lib/zookeeper
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HUE_PLUGINS_HOME: /usr/lib/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: JSVC_HOME: /usr/libexec/bigtop-utils
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HDFS_HOME: /usr/lib/hadoop-hdfs
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_MR1_HOME: /usr/lib/hadoop-0.20-mapreduce
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_MR2_HOME: /usr/lib/hadoop-mapreduce
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HADOOP_HOME: /usr/lib/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_IMPALA_HOME: /usr/lib/impala
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Logging to
    /var/log/cloudera-scm-agent/cloudera-scm-agent.log
    Traceback (most recent call last):
    File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 2391, in <module>
    main()
    File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 2375, in main
    agent.start()
    File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 329, in start
    self.add_db_driver_jars()
    File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 1400, in
    add_db_driver_ja>>/usr/lib64/cmf/agent/src/cmf/agent.py:31:
    DeprecationWarning: the sha module is deprecated; use the hashlib module
    instead
    import sha
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO SCM Agent
    Version: 4.6.0
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Using
    directory: /var/run/cloudera-scm-agent
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Using
    supervisor binary path:
    /usr/lib64/cmf/agent/src/cmf/../../build/env/bin/supervisord
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Adding env vars
    that start with CMF_AGENT_
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: HIVE_DEFAULT_XML:
    /etc/hive/conf.dist/hive-default.xml
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: MGMT_HOME: /usr/share/cmf
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HTTPFS_HOME: /usr/lib/hadoop-httpfs
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_FLUME_HOME: /usr/lib/flume-ng
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HBASE_HOME: /usr/lib/hbase
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HUE_HOME: /usr/share/hue
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: TOMCAT_HOME: /usr/lib/bigtop-tomcat
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_PIG_HOME: /usr/lib/pig
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HIVE_HOME: /usr/lib/hive
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_YARN_HOME: /usr/lib/hadoop-yarn
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HADOOP_BIN: /usr/bin/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_OOZIE_HOME: /usr/lib/oozie
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_ZOOKEEPER_HOME: /usr/lib/zookeeper
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HUE_PLUGINS_HOME: /usr/lib/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: JSVC_HOME: /usr/libexec/bigtop-utils
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HDFS_HOME: /usr/lib/hadoop-hdfs
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_MR1_HOME: /usr/lib/hadoop-0.20-mapreduce
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_MR2_HOME: /usr/lib/hadoop-mapreduce
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HADOOP_HOME: /usr/lib/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_IMPALA_HOME: /usr/lib/impala
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Logging to
    /var/log/cloudera-scm-agent/cloudera-scm-agent.log
    Traceback (most recent call last):
    File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 2391, in <module>
    main()
    File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 2375, in main
    agent.start()
    File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 329, in start
    self.add_db_driver_jars()
    File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 1400, in
    add_db_driver_jars
    for jar in (mysql_jar, oracle_jar, postgresql_jar):
    UnboundLocalError: local variable 'postgresql_jar' referenced before
    assignment
    rs
    for jar in (mysql_jar, oracle_jar, postgresql_jar):
    UnboundLocalError: local variable 'postgresql_jar' referenced before
    assignment
    *END (0) *
    *BEGIN tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.log |
    sed 's/^/>>/' *
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Deleting process
    1014-mapreduce-TASKTRACKER
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Triggering
    supervisord update.
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Retiring process
    1014-mapreduce-TASKTRACKER
    [07/Jun/2013 22:59:32 +0000] 695 MonitorDaemon-Reporter
    throttling_logger ERROR Error sending messages to firehose:
    mgmt1-HOSTMONITOR-d889e476015d035afbfed2002d76de2d
    Traceback (most recent call last):
    File "/usr/lib64/cmf/agent/src/cmf/monitor/firehose.py", line 67, in
    send
    self._requestor.request('sendAgentMessages', dict(messages=messages))
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py",
    line 145, in request
    return self.issue_request(call_request, message_name, request_datum)
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py",
    line 256, in issue_request
    call_response = self.transceiver.transceive(call_request)
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py",
    line 485, in transceive
    result = self.read_framed_message()
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py",
    line 489, in read_framed_message
    response = self.conn.getresponse()
    File "/usr/lib64/python2.6/httplib.py", line 990, in getresponse
    response.begin()
    File "/usr/lib64/python2.6/httplib.py", line 391, in begin
    version, status, reason = self._read_status()
    File "/usr/lib64/python2.6/httplib.py", line 355, in _read_status
    raise BadStatusLine(line)
    BadStatusLine
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Stopping agent...
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO No cgrouped
    processes running; unmounting any cgroup roots
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO No processes are
    being managed; Supervisor will shutdown.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Shutting down
    supervisord, pid 715
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Successfully
    shutdown supervisord, pid 715
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE HTTP Server
    cherrypy._cpwsgi_server.CPWSGIServer(('dn6.lab10.adconion.com', 9000))
    shut down
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Stopped thread '_TimeoutMonitor'.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE HTTP Server
    cherrypy._cpwsgi_server.CPWSGIServer(('dn6.lab10.adconion.com', 9000))
    already shut down
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE No thread running for None.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus EXITING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus EXITED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Agent exiting;
    caught signal 15
    [07/Jun/2013 23:02:30 +0000] 12620 MainThread agent INFO No command line
    vars
    [07/Jun/2013 23:02:30 +0000] 12620 Dummy-1 agent INFO Stopping agent...
    [07/Jun/2013 23:20:25 +0000] 14136 MainThread agent INFO No command line
    vars
    [07/Jun/2013 23:20:25 +0000] 14136 Dummy-1 agent INFO Stopping agent...
    [07/Jun/2013 23:51:20 +0000] 15580 MainThread agent INFO No command line
    vars
    [07/Jun/2013 23:51:20 +0000] 15580 Dummy-1 agent INFO Stopping agent...
    [07/Jun/2013 23:53:21 +0000] 16382 MainThread agent INFO No command line
    vars
    [07/Jun/2013 23:53:21 +0000] 16382 Dummy-1 agent INFO Stopping agent...
    [08/Jun/2013 00:05:13 +0000] 17261 MainThread agent INFO No command line
    vars
    [08/Jun/2013 00:05:13 +0000] 17261 Dummy-1 agent INFO Stopping agent...
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO No command line
    vars
    [08/Jun/2013 00:43:29 +0000] 20804 Dummy-1 agent INFO Stopping agent...
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Deleting process
    1014-mapreduce-TASKTRACKER
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Triggering
    supervisord update.
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Retiring process
    1014-mapreduce-TASKTRACKER
    [07/Jun/2013 22:59:32 +0000] 695 MonitorDaemon-Reporter
    throttling_logger ERROR Error sending messages to firehose:
    mgmt1-HOSTMONITOR-d889e476015d035afbfed2002d76de2d
    Traceback (most recent call last):
    File "/usr/lib64/cmf/agent/src/cmf/monitor/firehose.py", line 67, in
    send
    self._requestor.request('sendAgentMessages', dict(messages=messages))
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py",
    line 145, in request
    return self.issue_request(call_request, message_name, request_datum)
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py",
    line 256, in issue_request
    call_response = self.transceiver.transceive(call_request)
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py",
    line 485, in transceive
    result = self.read_framed_message()
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py",
    line 489, in read_framed_message
    response = self.conn.getresponse()
    File "/usr/lib64/python2.6/httplib.py", line 990, in getresponse
    response.begin()
    File "/usr/lib64/python2.6/httplib.py", line 391, in begin
    version, status, reason = self._read_status()
    File "/usr/lib64/python2.6/httplib.py", line 355, in _read_status
    raise BadStatusLine(line)
    BadStatusLine
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Stopping agent...
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO No cgrouped
    processes running; unmounting any cgroup roots
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO No processes are
    being managed; Supervisor will shutdown.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Shutting down
    supervisord, pid 715
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Successfully
    shutdown supervisord, pid 715
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE HTTP Server
    cherrypy._cpwsgi_server.CPWSGIServer(('dn6.lab10.adconion.com', 9000))
    shut down
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Stopped thread '_TimeoutMonitor'.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE HTTP Server
    cherrypy._cpwsgi_server.CPWSGIServer(('dn6.lab10.adconion.com', 9000))
    already shut down
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE No thread running for None.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus EXITING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus EXITED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Agent exiting;
    caught signal 15
    [07/Jun/2013 23:02:30 +0000] 12620 MainThread agent INFO No command line
    vars
    [07/Jun/2013 23:02:30 +0000] 12620 Dummy-1 agent INFO Stopping agent...
    [07/Jun/2013 23:20:25 +0000] 14136 MainThread agent INFO No command line
    vars
    [07/Jun/2013 23:20:25 +0000] 14136 Dummy-1 agent INFO Stopping agent...
    [07/Jun/2013 23:51:20 +0000] 15580 MainThread agent INFO No command line
    vars
    [07/Jun/2013 23:51:20 +0000] 15580 Dummy-1 agent INFO Stopping agent...
    [07/Jun/2013 23:53:21 +0000] 16382 MainThread agent INFO No command line
    vars
    [07/Jun/2013 23:53:21 +0000] 16382 Dummy-1 agent INFO Stopping agent...
    [08/Jun/2013 00:05:13 +0000] 17261 MainThread agent INFO No command line
    vars
    [08/Jun/2013 00:05:13 +0000] 17261 Dummy-1 agent INFO Stopping agent...
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO No command line
    vars
    [08/Jun/2013 00:43:29 +0000] 20804 Dummy-1 agent INFO Stopping agent...
    *END (0) *
    end of agent logs.
    scm agent could not be started, giving up
    waiting for rollback request

    I hope you can help.

    Thanks,
    Ben
    On Friday, June 7, 2013 3:55:09 PM UTC-7, Benjamin Kim wrote:

    I just upgraded the CM server in EC2 to 4.6 from 4.5.2. I am trying to
    upgrade the node agents to the 4.6 using the wizard. I have 11 EC2 nodes in
    the cluster. When it gets to the Upgrade Hosts page, it cannot connect to
    any of the nodes in EC2. The names of all the nodes have been replaced with
    names like "i-4f792e33". I can see the original hosts with their correct
    names on the Hosts page. What can I do to fix this?

    Thanks,
    Ben

    --
    Thanks,
    Darren
  • Benjamin Kim at Jun 10, 2013 at 4:16 pm ⇧
    Hi Darren,

    I did something more drastic. I uninstalled CM and reinstalled CM4.6 from
    scratch. After doing this, it worked. Since the root directories on my
    m1.large EC2 instances don't have much disk space, I had to change the
    Cloudera Manager log and embedded PostgreSQL DB file locations to a much
    larger secondary drive. I modified the scripts, cloudera-scm-server and
    cloudera-scm-server-db, in the /etc/init.d directory directly by changing
    references to the data directory from /var/lib to /u00/lib and references
    to the log directory from /var/log to /var/lib. Is this the right place to
    do these changes? I don't see any of these options being configurable in
    Cloudera Manager itself.

    On another note, I'll be troubleshooting the cluster to see why all the
    health checks are bad. But, all the services seem to be working fine.

    Thanks,
    Ben
    On Sunday, June 9, 2013 10:50:46 PM UTC-7, Darren Lo wrote:

    Hi Ben,

    Sorry for the toruble, and thanks for finding this bug! I've reproduced
    the error locally and I will make sure this is fixed in 4.6.1.

    In the meantime, you can get past this bug by removing the export line for
    CLOUDERA_POSTGRESQL_JDBC_JAR from /etc/default/cloudera-scm-agent on all
    hosts. It isn't necessary anymore in 4.6. The agent code will correctly
    locate the jar in the cm lib dir by default (normally /usr/share/cmf/lib).

    Let me know if it works!

    Thanks,
    Darren


    On Fri, Jun 7, 2013 at 5:51 PM, Benjamin Kim <bbui...@gmail.com<javascript:>
    wrote:
    I was able to fix the problem by going into the scm db and modifying the
    hosts table to have the identifier also be the name. But now, I'm having
    another issue. The CM Agents will not start up.

    Here is the log trace returned.

    /tmp/scm_prepare_node.jChJxhLv
    using SSH_CLIENT to get the SCM hostname: 10.124.243.21 37046 22
    opening logging file descriptor
    Starting installation script... Acquiring installation lock...
    *BEGIN flock 4 *
    *END (0) *
    Detecting root privileges...
    effective UID is 0
    Detecting distribution...
    *BEGIN grep Tikanga /etc/redhat-release *
    *END (1) *
    *BEGIN grep 'CentOS release 5' /etc/redhat-release *
    *END (1) *
    *BEGIN grep 'Scientific Linux release 5' /etc/redhat-release *
    *END (1) *
    *BEGIN grep Santiago /etc/redhat-release *
    *END (1) *
    *BEGIN grep 'CentOS Linux release 6' /etc/redhat-release *
    *END (1) *
    *BEGIN grep 'CentOS release 6' /etc/redhat-release *
    CentOS release 6.3 (Final)
    *END (0) *
    /etc/redhat-release ==> CentOS 6
    Detecting Cloudera Manager Server...
    *BEGIN host -t PTR 10.124.243.21 *
    21.243.124.10.in-addr.arpa domain name pointer
    ip-10-124-243-21.ec2.internal.
    *END (0) *
    using ip-10-124-243-21.ec2.internal as scm server hostname
    *BEGIN which python *
    /usr/bin/python
    *END (0) *
    *BEGIN python -c 'import socket; import sys; s =
    socket.socket(socket.AF_INET); s.settimeout(5.0); s.connect((sys.argv[1],
    int(sys.argv[2]))); s.close();' ip-10-124-243-21.ec2.internal 7182 *
    *END (0) *
    *BEGIN which wget *
    /usr/bin/wget
    *END (0) *
    *BEGIN wget -qO- -T 1 -t 1
    http://169.254.169.254/latest/meta-data/public-hostname && /bin/echo *
    ec2-107-21-133-38.compute-1.amazonaws.com
    *END (0) *
    *BEGIN wget -qO- -T 1 -t 1
    http://169.254.169.254/latest/meta-data/instance-id && /bin/echo *
    i-05653279
    *END (0) *
    Performing cloud-specific configuration...
    *BEGIN /sbin/chkconfig iptables *
    *END (1) *
    *BEGIN /sbin/service iptables status | grep Chain *
    Chain INPUT (policy ACCEPT)
    Chain FORWARD (policy ACCEPT)
    Chain OUTPUT (policy ACCEPT)
    Chain RH-Firewall-1-INPUT (2 references)
    *END (0) *
    *BEGIN /sbin/service iptables stop *
    iptables: Flushing firewall rules: [60G[ [0;32m OK [0;39m]
    iptables: Setting chains to policy ACCEPT: filter [60G[ [0;32m OK [0;39m]
    iptables: Unloading modules: [60G[ [0;32m OK [0;39m]
    *END (0) *
    *BEGIN getenforce *
    Disabled
    *END (0) *
    *BEGIN grep -e ^SELINUX=enforcing /etc/selinux/config *
    *END (1) *
    *BEGIN df -h | grep -o -E "^/dev/[^[:space:]]*" *
    /dev/xvde1
    /dev/md0
    *END (0) *
    *BEGIN awk 'FNR > 2 {print $NF}' /proc/partitions *
    xvdf
    xvdf1
    xvdf2
    xvdg
    xvdg1
    xvdg2
    xvde
    xvde1
    md0
    *END (0) *
    Installing package repositories...
    validating format of repository file
    /tmp/scm_prepare_node.jChJxhLv/repos/rhel6/cloudera-manager.repo
    installing repository file
    /tmp/scm_prepare_node.jChJxhLv/repos/rhel6/cloudera-manager.repo
    repository file
    /tmp/scm_prepare_node.jChJxhLv/repos/rhel6/cloudera-manager.repo installed
    Refreshing package metadata...
    *BEGIN yum clean all *
    Loaded plugins: fastestmirror, presto
    Cleaning repos: base cloudera-cdh4 cloudera-impala cloudera-manager epel
    extras
    : updates
    Cleaning up Everything
    0 delta-package files removed, by presto
    *END (0) *
    *BEGIN rm -Rf /var/cache/yum/x86_64 *
    *END (0) *
    *BEGIN yum makecache *
    Loaded plugins: fastestmirror, presto
    Metadata Cache Created
    *END (0) *
    Installing jdk package...
    *BEGIN yum list installed jdk *
    Loaded plugins: fastestmirror, presto
    Installed Packages
    jdk.x86_64 2000:1.6.0_31-fcs @cloudera-manager
    *END (0) *
    *BEGIN echo jdk cloudera-manager-agent cloudera-manager-daemons | grep
    jdk *
    jdk cloudera-manager-agent cloudera-manager-daemons
    *END (0) *
    *BEGIN yum info jdk *
    Loaded plugins: fastestmirror, presto
    Determining fastest mirrors
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.fedoraproject.org
    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Installed Packages
    Name : jdk
    Arch : x86_64
    Epoch : 2000
    Version : 1.6.0_31
    Release : fcs
    Size : 143 M
    Repo : installed
    From repo : cloudera-manager
    Summary : Java(TM) Platform Standard Edition Development Kit
    URL : http://java.sun.com/
    License : Copyright (c) 2011, Oracle and/or its affiliates. All rights
    : reserved. Also under other license(s) as shown at the Description
    : field.
    Description : The Java Platform Standard Edition Development Kit (JDK)
    includes
    : both the runtime environment (Java virtual machine, the Java
    : platform classes and supporting files) and development tools
    : (compilers, debuggers, tool libraries and other tools).
    :
    : The JDK is a development environment for building applications,
    : applets and components that can be deployed with the Java Platform
    : Standard Edition Runtime Environment.

    *END (0) *
    *BEGIN yum -y install jdk.x86_64 *
    Loaded plugins: fastestmirror, presto
    Loading mirror speeds from cached hostfile
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.fedoraproject.org
    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Setting up Install Process
    Package 2000:jdk-1.6.0_31-fcs.x86_64 already installed and latest version
    Nothing to do
    *END (0) *
    remote package jdk installed
    Installing cloudera-manager-agent package...
    *BEGIN yum list installed cloudera-manager-agent *
    Loaded plugins: fastestmirror, presto
    Installed Packages
    cloudera-manager-agent.x86_64 4.6.0-1.cm460.p0.140 @cloudera-manager
    *END (0) *
    *BEGIN echo jdk cloudera-manager-agent cloudera-manager-daemons | grep
    cloudera-manager-agent *
    *END (0) *
    jdk cloudera-manager-agent cloudera-manager-daemons
    *BEGIN yum info cloudera-manager-agent *
    Loaded plugins: fastestmirror, presto
    Loading mirror speeds from cached hostfile
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.fedoraproject.org
    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Installed Packages
    Name : cloudera-manager-agent
    Arch : x86_64
    Version : 4.6.0
    Release : 1.cm460.p0.140
    Size : 25 M
    Repo : installed
    From repo : cloudera-manager
    Summary : The Cloudera Manager Agent
    URL : http://www.cloudera.com
    License : Proprietary
    Description : The Cloudera Manager Agent.
    :
    : The Agent is deployed to machines running services managed by
    : Cloudera Manager.

    *END (0) *
    Version : 4.6.0
    Release : 1.cm460.p0.140
    *BEGIN yum -y install cloudera-manager-agent *
    Loaded plugins: fastestmirror, presto
    Loading mirror speeds from cached hostfile
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.fedoraproject.org
    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Setting up Install Process
    Package cloudera-manager-agent-4.6.0-1.cm460.p0.140.x86_64 already
    installed and latest version
    Nothing to do
    *END (0) *
    remote package cloudera-manager-agent installed
    Installing cloudera-manager-daemons package...
    *BEGIN yum list installed cloudera-manager-daemons *
    Loaded plugins: fastestmirror, presto
    Installed Packages
    cloudera-manager-daemons.x86_64 4.6.0-1.cm460.p0.140 @cloudera-manager
    *END (0) *
    *BEGIN echo jdk cloudera-manager-agent cloudera-manager-daemons | grep
    cloudera-manager-daemons *
    jdk cloudera-manager-agent cloudera-manager-daemons
    *END (0) *
    *BEGIN yum info cloudera-manager-daemons *
    Loaded plugins: fastestmirror, presto
    Loading mirror speeds from cached hostfile
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.fedoraproject.org
    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Installed Packages
    Name : cloudera-manager-daemons
    Arch : x86_64
    Version : 4.6.0
    Release : 1.cm460.p0.140
    Size : 175 M
    Repo : installed
    From repo : cloudera-manager
    Summary : Provides daemons for monitoring Hadoop and related tools.
    URL : http://www.cloudera.com
    License : Proprietary
    Description : This package includes daemons for monitoring and managing
    Hadoop.

    *END (0) *
    Version : 4.6.0
    Release : 1.cm460.p0.140
    *BEGIN yum -y install cloudera-manager-daemons *
    Loaded plugins: fastestmirror, presto
    Loading mirror speeds from cached hostfile
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.fedoraproject.org
    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Setting up Install Process
    Package cloudera-manager-daemons-4.6.0-1.cm460.p0.140.x86_64 already
    installed and latest version
    Nothing to do
    *END (0) *
    remote package cloudera-manager-daemons installed
    Configuring Cloudera Manager Agent...
    *BEGIN sed -e 's/\(CMF_AGENT_ARGS=\).*/\1"--host_id i-05653279"/' -i
    /etc/default/cloudera-scm-agent *
    *END (0) *
    *BEGIN grep server_host=ip-10-124-243-21.ec2.internal
    /etc/cloudera-scm-agent/config.ini *
    server_host=ip-10-124-243-21.ec2.internal
    *END (0) *
    scm agent is already configured
    Starting Cloudera Manager Agent...
    *BEGIN /sbin/service cloudera-scm-agent status | grep running *
    *END (1) *
    *BEGIN /sbin/service cloudera-scm-agent start *
    Starting cloudera-scm-agent: [60G[ [0;31mFAILED [0;39m]
    *END (1) *
    agent logs:
    *BEGIN tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.out |
    sed 's/^/>>/' *
    /usr/lib64/cmf/agent/src/cmf/agent.py:31: DeprecationWarning: the sha
    module is deprecated; use the hashlib module instead
    import sha
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO SCM Agent
    Version: 4.6.0
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Using
    directory: /var/run/cloudera-scm-agent
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Using
    supervisor binary path:
    /usr/lib64/cmf/agent/src/cmf/../../build/env/bin/supervisord
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Adding env
    vars that start with CMF_AGENT_
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: HIVE_DEFAULT_XML:
    /etc/hive/conf.dist/hive-default.xml
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: MGMT_HOME: /usr/share/cmf
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HTTPFS_HOME: /usr/lib/hadoop-httpfs
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_FLUME_HOME: /usr/lib/flume-ng
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HBASE_HOME: /usr/lib/hbase
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HUE_HOME: /usr/share/hue
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: TOMCAT_HOME: /usr/lib/bigtop-tomcat
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_PIG_HOME: /usr/lib/pig
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HIVE_HOME: /usr/lib/hive
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_YARN_HOME: /usr/lib/hadoop-yarn
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HADOOP_BIN: /usr/bin/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_OOZIE_HOME: /usr/lib/oozie
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_ZOOKEEPER_HOME: /usr/lib/zookeeper
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HUE_PLUGINS_HOME: /usr/lib/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: JSVC_HOME: /usr/libexec/bigtop-utils
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HDFS_HOME: /usr/lib/hadoop-hdfs
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_MR1_HOME: /usr/lib/hadoop-0.20-mapreduce
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_MR2_HOME: /usr/lib/hadoop-mapreduce
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HADOOP_HOME: /usr/lib/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_IMPALA_HOME: /usr/lib/impala
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Logging to
    /var/log/cloudera-scm-agent/cloudera-scm-agent.log
    Traceback (most recent call last):
    File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 2391, in <module>
    main()
    File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 2375, in main
    agent.start()
    File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 329, in start
    self.add_db_driver_jars()
    File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 1400, in
    add_db_driver_ja>>/usr/lib64/cmf/agent/src/cmf/agent.py:31:
    DeprecationWarning: the sha module is deprecated; use the hashlib module
    instead
    import sha
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO SCM Agent
    Version: 4.6.0
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Using
    directory: /var/run/cloudera-scm-agent
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Using
    supervisor binary path:
    /usr/lib64/cmf/agent/src/cmf/../../build/env/bin/supervisord
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Adding env
    vars that start with CMF_AGENT_
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: HIVE_DEFAULT_XML:
    /etc/hive/conf.dist/hive-default.xml
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: MGMT_HOME: /usr/share/cmf
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HTTPFS_HOME: /usr/lib/hadoop-httpfs
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_FLUME_HOME: /usr/lib/flume-ng
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HBASE_HOME: /usr/lib/hbase
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HUE_HOME: /usr/share/hue
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: TOMCAT_HOME: /usr/lib/bigtop-tomcat
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_PIG_HOME: /usr/lib/pig
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HIVE_HOME: /usr/lib/hive
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_YARN_HOME: /usr/lib/hadoop-yarn
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HADOOP_BIN: /usr/bin/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_OOZIE_HOME: /usr/lib/oozie
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_ZOOKEEPER_HOME: /usr/lib/zookeeper
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HUE_PLUGINS_HOME: /usr/lib/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: JSVC_HOME: /usr/libexec/bigtop-utils
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HDFS_HOME: /usr/lib/hadoop-hdfs
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_MR1_HOME: /usr/lib/hadoop-0.20-mapreduce
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_MR2_HOME: /usr/lib/hadoop-mapreduce
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HADOOP_HOME: /usr/lib/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_IMPALA_HOME: /usr/lib/impala
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Logging to
    /var/log/cloudera-scm-agent/cloudera-scm-agent.log
    Traceback (most recent call last):
    File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 2391, in <module>
    main()
    File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 2375, in main
    agent.start()
    File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 329, in start
    self.add_db_driver_jars()
    File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 1400, in
    add_db_driver_jars
    for jar in (mysql_jar, oracle_jar, postgresql_jar):
    UnboundLocalError: local variable 'postgresql_jar' referenced before
    assignment
    rs
    for jar in (mysql_jar, oracle_jar, postgresql_jar):
    UnboundLocalError: local variable 'postgresql_jar' referenced before
    assignment
    *END (0) *
    *BEGIN tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.log |
    sed 's/^/>>/' *
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Deleting process
    1014-mapreduce-TASKTRACKER
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Triggering
    supervisord update.
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Retiring process
    1014-mapreduce-TASKTRACKER
    [07/Jun/2013 22:59:32 +0000] 695 MonitorDaemon-Reporter
    throttling_logger ERROR Error sending messages to firehose:
    mgmt1-HOSTMONITOR-d889e476015d035afbfed2002d76de2d
    Traceback (most recent call last):
    File "/usr/lib64/cmf/agent/src/cmf/monitor/firehose.py", line 67, in
    send
    self._requestor.request('sendAgentMessages', dict(messages=messages))
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py",
    line 145, in request
    return self.issue_request(call_request, message_name, request_datum)
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py",
    line 256, in issue_request
    call_response = self.transceiver.transceive(call_request)
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py",
    line 485, in transceive
    result = self.read_framed_message()
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py",
    line 489, in read_framed_message
    response = self.conn.getresponse()
    File "/usr/lib64/python2.6/httplib.py", line 990, in getresponse
    response.begin()
    File "/usr/lib64/python2.6/httplib.py", line 391, in begin
    version, status, reason = self._read_status()
    File "/usr/lib64/python2.6/httplib.py", line 355, in _read_status
    raise BadStatusLine(line)
    BadStatusLine
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Stopping
    agent...
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO No cgrouped
    processes running; unmounting any cgroup roots
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO No processes are
    being managed; Supervisor will shutdown.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Shutting down
    supervisord, pid 715
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Successfully
    shutdown supervisord, pid 715
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE HTTP Server
    cherrypy._cpwsgi_server.CPWSGIServer(('dn6.lab10.adconion.com', 9000))
    shut down
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Stopped thread '_TimeoutMonitor'.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE HTTP Server
    cherrypy._cpwsgi_server.CPWSGIServer(('dn6.lab10.adconion.com', 9000))
    already shut down
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE No thread running for None.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus EXITING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus EXITED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Agent exiting;
    caught signal 15
    [07/Jun/2013 23:02:30 +0000] 12620 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:02:30 +0000] 12620 Dummy-1 agent INFO Stopping agent...
    [07/Jun/2013 23:20:25 +0000] 14136 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:20:25 +0000] 14136 Dummy-1 agent INFO Stopping agent...
    [07/Jun/2013 23:51:20 +0000] 15580 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:51:20 +0000] 15580 Dummy-1 agent INFO Stopping agent...
    [07/Jun/2013 23:53:21 +0000] 16382 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:53:21 +0000] 16382 Dummy-1 agent INFO Stopping agent...
    [08/Jun/2013 00:05:13 +0000] 17261 MainThread agent INFO No command
    line vars
    [08/Jun/2013 00:05:13 +0000] 17261 Dummy-1 agent INFO Stopping agent...
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO No command
    line vars
    [08/Jun/2013 00:43:29 +0000] 20804 Dummy-1 agent INFO Stopping agent...
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Deleting process
    1014-mapreduce-TASKTRACKER
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Triggering
    supervisord update.
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Retiring process
    1014-mapreduce-TASKTRACKER
    [07/Jun/2013 22:59:32 +0000] 695 MonitorDaemon-Reporter
    throttling_logger ERROR Error sending messages to firehose:
    mgmt1-HOSTMONITOR-d889e476015d035afbfed2002d76de2d
    Traceback (most recent call last):
    File "/usr/lib64/cmf/agent/src/cmf/monitor/firehose.py", line 67, in
    send
    self._requestor.request('sendAgentMessages', dict(messages=messages))
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py",
    line 145, in request
    return self.issue_request(call_request, message_name, request_datum)
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py",
    line 256, in issue_request
    call_response = self.transceiver.transceive(call_request)
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py",
    line 485, in transceive
    result = self.read_framed_message()
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py",
    line 489, in read_framed_message
    response = self.conn.getresponse()
    File "/usr/lib64/python2.6/httplib.py", line 990, in getresponse
    response.begin()
    File "/usr/lib64/python2.6/httplib.py", line 391, in begin
    version, status, reason = self._read_status()
    File "/usr/lib64/python2.6/httplib.py", line 355, in _read_status
    raise BadStatusLine(line)
    BadStatusLine
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Stopping
    agent...
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO No cgrouped
    processes running; unmounting any cgroup roots
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO No processes are
    being managed; Supervisor will shutdown.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Shutting down
    supervisord, pid 715
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Successfully
    shutdown supervisord, pid 715
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE HTTP Server
    cherrypy._cpwsgi_server.CPWSGIServer(('dn6.lab10.adconion.com', 9000))
    shut down
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Stopped thread '_TimeoutMonitor'.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE HTTP Server
    cherrypy._cpwsgi_server.CPWSGIServer(('dn6.lab10.adconion.com', 9000))
    already shut down
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE No thread running for None.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus EXITING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus EXITED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Agent exiting;
    caught signal 15
    [07/Jun/2013 23:02:30 +0000] 12620 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:02:30 +0000] 12620 Dummy-1 agent INFO Stopping agent...
    [07/Jun/2013 23:20:25 +0000] 14136 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:20:25 +0000] 14136 Dummy-1 agent INFO Stopping agent...
    [07/Jun/2013 23:51:20 +0000] 15580 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:51:20 +0000] 15580 Dummy-1 agent INFO Stopping agent...
    [07/Jun/2013 23:53:21 +0000] 16382 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:53:21 +0000] 16382 Dummy-1 agent INFO Stopping agent...
    [08/Jun/2013 00:05:13 +0000] 17261 MainThread agent INFO No command
    line vars
    [08/Jun/2013 00:05:13 +0000] 17261 Dummy-1 agent INFO Stopping agent...
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO No command
    line vars
    [08/Jun/2013 00:43:29 +0000] 20804 Dummy-1 agent INFO Stopping agent...
    *END (0) *
    end of agent logs.
    scm agent could not be started, giving up
    waiting for rollback request

    I hope you can help.

    Thanks,
    Ben
    On Friday, June 7, 2013 3:55:09 PM UTC-7, Benjamin Kim wrote:

    I just upgraded the CM server in EC2 to 4.6 from 4.5.2. I am trying to
    upgrade the node agents to the 4.6 using the wizard. I have 11 EC2 nodes in
    the cluster. When it gets to the Upgrade Hosts page, it cannot connect to
    any of the nodes in EC2. The names of all the nodes have been replaced with
    names like "i-4f792e33". I can see the original hosts with their
    correct names on the Hosts page. What can I do to fix this?

    Thanks,
    Ben

    --
    Thanks,
    Darren
  • Darren Lo at Jun 10, 2013 at 5:30 pm ⇧
    Hi Ben,

    The embedded postgresql is intended to be a "quick-start" option and is not
    configurable. We assume that if someone wants to customize their database,
    then they will just set one up on their own.

    I expect that CM code won't correctly detect the embedded postgresql DB if
    it's in a different location. It could even error out. If the database is
    working fine, then go ahead and continue to use it, but treat it as a
    non-embedded postgresql. This means you should remove or rename
    /etc/cloudera-scm-server/db.mgmt.properties, and don't expect embedded db
    features such as automatic user creation for management services and Hive
    (though we'll still create the tables).

    If you really want to try and use the embedded db in a custom location, you
    might want to make a simlink from the normal
    /var/lib/cloudera-scm-server-db/data to your new dir. This will probably
    work, but I can't guarantee anything here.

    Thanks,
    Darren

    On Mon, Jun 10, 2013 at 9:16 AM, Benjamin Kim wrote:

    Hi Darren,

    I did something more drastic. I uninstalled CM and reinstalled CM4.6 from
    scratch. After doing this, it worked. Since the root directories on my
    m1.large EC2 instances don't have much disk space, I had to change the
    Cloudera Manager log and embedded PostgreSQL DB file locations to a much
    larger secondary drive. I modified the scripts, cloudera-scm-server and
    cloudera-scm-server-db, in the /etc/init.d directory directly by changing
    references to the data directory from /var/lib to /u00/lib and references
    to the log directory from /var/log to /var/lib. Is this the right place to
    do these changes? I don't see any of these options being configurable in
    Cloudera Manager itself.

    On another note, I'll be troubleshooting the cluster to see why all the
    health checks are bad. But, all the services seem to be working fine.

    Thanks,
    Ben

    On Sunday, June 9, 2013 10:50:46 PM UTC-7, Darren Lo wrote:

    Hi Ben,

    Sorry for the toruble, and thanks for finding this bug! I've reproduced
    the error locally and I will make sure this is fixed in 4.6.1.

    In the meantime, you can get past this bug by removing the export line
    for CLOUDERA_POSTGRESQL_JDBC_JAR from /etc/default/cloudera-**scm-agent
    on all hosts. It isn't necessary anymore in 4.6. The agent code will
    correctly locate the jar in the cm lib dir by default (normally
    /usr/share/cmf/lib).

    Let me know if it works!

    Thanks,
    Darren

    On Fri, Jun 7, 2013 at 5:51 PM, Benjamin Kim wrote:

    I was able to fix the problem by going into the scm db and modifying the
    hosts table to have the identifier also be the name. But now, I'm having
    another issue. The CM Agents will not start up.

    Here is the log trace returned.

    /tmp/scm_prepare_node.**jChJxhLv
    using SSH_CLIENT to get the SCM hostname: 10.124.243.21 37046 22
    opening logging file descriptor
    Starting installation script... Acquiring installation lock...
    *BEGIN flock 4 *
    *END (0) *
    Detecting root privileges...
    effective UID is 0
    Detecting distribution...
    *BEGIN grep Tikanga /etc/redhat-release *
    *END (1) *
    *BEGIN grep 'CentOS release 5' /etc/redhat-release *
    *END (1) *
    *BEGIN grep 'Scientific Linux release 5' /etc/redhat-release *
    *END (1) *
    *BEGIN grep Santiago /etc/redhat-release *
    *END (1) *
    *BEGIN grep 'CentOS Linux release 6' /etc/redhat-release *
    *END (1) *
    *BEGIN grep 'CentOS release 6' /etc/redhat-release *
    CentOS release 6.3 (Final)
    *END (0) *
    /etc/redhat-release ==> CentOS 6
    Detecting Cloudera Manager Server...
    *BEGIN host -t PTR 10.124.243.21 *
    21.243.124.10.in-addr.arpa domain name pointer
    ip-10-124-243-21.ec2.internal.**
    *END (0) *
    using ip-10-124-243-21.ec2.internal as scm server hostname
    *BEGIN which python *
    /usr/bin/python
    *END (0) *
    *BEGIN python -c 'import socket; import sys; s =
    socket.socket(socket.AF_INET); s.settimeout(5.0); s.connect((sys.argv[1],
    int(sys.argv[2]))); s.close();' ip-10-124-243-21.ec2.internal 7182 *
    *END (0) *
    *BEGIN which wget *
    /usr/bin/wget
    *END (0) *
    *BEGIN wget -qO- -T 1 -t 1 http://169.254.169.254/latest/
    meta-data/public-hostname && /bin/echo *
    ec2-107-21-133-38.compute-1.**amazonaws.com<http://ec2-107-21-133-38.compute-1.amazonaws.com>

    *END (0) *
    *BEGIN wget -qO- -T 1 -t 1 http://169.254.169.254/latest/
    meta-data/instance-id && /bin/echo *
    i-05653279
    *END (0) *
    Performing cloud-specific configuration...
    *BEGIN /sbin/chkconfig iptables *
    *END (1) *
    *BEGIN /sbin/service iptables status | grep Chain *
    Chain INPUT (policy ACCEPT)
    Chain FORWARD (policy ACCEPT)
    Chain OUTPUT (policy ACCEPT)
    Chain RH-Firewall-1-INPUT (2 references)
    *END (0) *
    *BEGIN /sbin/service iptables stop *
    iptables: Flushing firewall rules: [60G[ [0;32m OK [0;39m]
    iptables: Setting chains to policy ACCEPT: filter [60G[ [0;32m OK
    [0;39m]
    iptables: Unloading modules: [60G[ [0;32m OK [0;39m]
    *END (0) *
    *BEGIN getenforce *
    Disabled
    *END (0) *
    *BEGIN grep -e ^SELINUX=enforcing /etc/selinux/config *
    *END (1) *
    *BEGIN df -h | grep -o -E "^/dev/[^[:space:]]*" *
    /dev/xvde1
    /dev/md0
    *END (0) *
    *BEGIN awk 'FNR > 2 {print $NF}' /proc/partitions *
    xvdf
    xvdf1
    xvdf2
    xvdg
    xvdg1
    xvdg2
    xvde
    xvde1
    md0
    *END (0) *
    Installing package repositories...
    validating format of repository file /tmp/scm_prepare_node.**
    jChJxhLv/repos/rhel6/cloudera-**manager.repo
    installing repository file /tmp/scm_prepare_node.**
    jChJxhLv/repos/rhel6/cloudera-**manager.repo
    repository file /tmp/scm_prepare_node.**jChJxhLv/repos/rhel6/cloudera-**manager.repo
    installed
    Refreshing package metadata...
    *BEGIN yum clean all *
    Loaded plugins: fastestmirror, presto
    Cleaning repos: base cloudera-cdh4 cloudera-impala cloudera-manager epel
    extras
    : updates
    Cleaning up Everything
    0 delta-package files removed, by presto
    *END (0) *
    *BEGIN rm -Rf /var/cache/yum/x86_64 *
    *END (0) *
    *BEGIN yum makecache *
    Loaded plugins: fastestmirror, presto
    Metadata Cache Created
    *END (0) *
    Installing jdk package...
    *BEGIN yum list installed jdk *
    Loaded plugins: fastestmirror, presto
    Installed Packages
    jdk.x86_64 2000:1.6.0_31-fcs @cloudera-manager
    *END (0) *
    *BEGIN echo jdk cloudera-manager-agent cloudera-manager-daemons | grep
    jdk *
    jdk cloudera-manager-agent cloudera-manager-daemons
    *END (0) *
    *BEGIN yum info jdk *
    Loaded plugins: fastestmirror, presto
    Determining fastest mirrors
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.**fedoraproject.org<http://s3-mirror-us-east-1.fedoraproject.org>

    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Installed Packages
    Name : jdk
    Arch : x86_64
    Epoch : 2000
    Version : 1.6.0_31
    Release : fcs
    Size : 143 M
    Repo : installed
    From repo : cloudera-manager
    Summary : Java(TM) Platform Standard Edition Development Kit
    URL : http://java.sun.com/
    License : Copyright (c) 2011, Oracle and/or its affiliates. All rights
    : reserved. Also under other license(s) as shown at the Description
    : field.
    Description : The Java Platform Standard Edition Development Kit (JDK)
    includes
    : both the runtime environment (Java virtual machine, the Java
    : platform classes and supporting files) and development tools
    : (compilers, debuggers, tool libraries and other tools).
    :
    : The JDK is a development environment for building applications,
    : applets and components that can be deployed with the Java Platform
    : Standard Edition Runtime Environment.

    *END (0) *
    *BEGIN yum -y install jdk.x86_64 *
    Loaded plugins: fastestmirror, presto
    Loading mirror speeds from cached hostfile
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.**fedoraproject.org<http://s3-mirror-us-east-1.fedoraproject.org>

    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Setting up Install Process
    Package 2000:jdk-1.6.0_31-fcs.x86_64 already installed and latest
    version
    Nothing to do
    *END (0) *
    remote package jdk installed
    Installing cloudera-manager-agent package...
    *BEGIN yum list installed cloudera-manager-agent *
    Loaded plugins: fastestmirror, presto
    Installed Packages
    cloudera-manager-agent.x86_64 4.6.0-1.cm460.p0.140 @cloudera-manager
    *END (0) *
    *BEGIN echo jdk cloudera-manager-agent cloudera-manager-daemons | grep
    cloudera-manager-agent *
    *END (0) *
    jdk cloudera-manager-agent cloudera-manager-daemons
    *BEGIN yum info cloudera-manager-agent *
    Loaded plugins: fastestmirror, presto
    Loading mirror speeds from cached hostfile
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.**fedoraproject.org<http://s3-mirror-us-east-1.fedoraproject.org>

    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Installed Packages
    Name : cloudera-manager-agent
    Arch : x86_64
    Version : 4.6.0
    Release : 1.cm460.p0.140
    Size : 25 M
    Repo : installed
    From repo : cloudera-manager
    Summary : The Cloudera Manager Agent
    URL : http://www.cloudera.com
    License : Proprietary
    Description : The Cloudera Manager Agent.
    :
    : The Agent is deployed to machines running services managed by
    : Cloudera Manager.

    *END (0) *
    Version : 4.6.0
    Release : 1.cm460.p0.140
    *BEGIN yum -y install cloudera-manager-agent *
    Loaded plugins: fastestmirror, presto
    Loading mirror speeds from cached hostfile
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.**fedoraproject.org<http://s3-mirror-us-east-1.fedoraproject.org>

    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Setting up Install Process
    Package cloudera-manager-agent-4.6.0-**1.cm460.p0.140.x86_64 already
    installed and latest version
    Nothing to do
    *END (0) *
    remote package cloudera-manager-agent installed
    Installing cloudera-manager-daemons package...
    *BEGIN yum list installed cloudera-manager-daemons *
    Loaded plugins: fastestmirror, presto
    Installed Packages
    cloudera-manager-daemons.x86_**64 4.6.0-1.cm460.p0.140
    @cloudera-manager
    *END (0) *
    *BEGIN echo jdk cloudera-manager-agent cloudera-manager-daemons | grep
    cloudera-manager-daemons *
    jdk cloudera-manager-agent cloudera-manager-daemons
    *END (0) *
    *BEGIN yum info cloudera-manager-daemons *
    Loaded plugins: fastestmirror, presto
    Loading mirror speeds from cached hostfile
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.**fedoraproject.org<http://s3-mirror-us-east-1.fedoraproject.org>

    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Installed Packages
    Name : cloudera-manager-daemons
    Arch : x86_64
    Version : 4.6.0
    Release : 1.cm460.p0.140
    Size : 175 M
    Repo : installed
    From repo : cloudera-manager
    Summary : Provides daemons for monitoring Hadoop and related tools.
    URL : http://www.cloudera.com
    License : Proprietary
    Description : This package includes daemons for monitoring and managing
    Hadoop.

    *END (0) *
    Version : 4.6.0
    Release : 1.cm460.p0.140
    *BEGIN yum -y install cloudera-manager-daemons *
    Loaded plugins: fastestmirror, presto
    Loading mirror speeds from cached hostfile
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.**fedoraproject.org<http://s3-mirror-us-east-1.fedoraproject.org>

    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Setting up Install Process
    Package cloudera-manager-daemons-4.6.**0-1.cm460.p0.140.x86_64 already
    installed and latest version
    Nothing to do
    *END (0) *
    remote package cloudera-manager-daemons installed
    Configuring Cloudera Manager Agent...
    *BEGIN sed -e 's/\(CMF_AGENT_ARGS=\).*/\1"--host_id i-05653279"/' -i
    /etc/default/cloudera-scm-agent *
    *END (0) *
    *BEGIN grep server_host=ip-10-124-243-21.ec2.internal
    /etc/cloudera-scm-agent/config.ini *
    server_host=ip-10-124-243-21.**ec2.internal
    *END (0) *
    scm agent is already configured
    Starting Cloudera Manager Agent...
    *BEGIN /sbin/service cloudera-scm-agent status | grep running *
    *END (1) *
    *BEGIN /sbin/service cloudera-scm-agent start *
    Starting cloudera-scm-agent: [60G[ [0;31mFAILED [0;39m]
    *END (1) *
    agent logs:
    *BEGIN tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.out |
    sed 's/^/>>/' *
    /usr/lib64/cmf/agent/src/**cmf/agent.py:31: DeprecationWarning: the
    sha module is deprecated; use the hashlib module instead
    import sha
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO SCM Agent
    Version: 4.6.0
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Using
    directory: /var/run/cloudera-scm-agent
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Using
    supervisor binary path: /usr/lib64/cmf/agent/src/cmf/.**
    ./../build/env/bin/**supervisord
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Adding env
    vars that start with CMF_AGENT_
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: HIVE_DEFAULT_XML: /etc/hive/conf.dist/hive-
    **default.xml
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: MGMT_HOME: /usr/share/cmf
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HTTPFS_HOME: /usr/lib/hadoop-httpfs
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_FLUME_HOME: /usr/lib/flume-ng
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HBASE_HOME: /usr/lib/hbase
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HUE_HOME: /usr/share/hue
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: TOMCAT_HOME: /usr/lib/bigtop-tomcat
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_PIG_HOME: /usr/lib/pig
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HIVE_HOME: /usr/lib/hive
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_YARN_HOME: /usr/lib/hadoop-yarn
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HADOOP_BIN: /usr/bin/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_OOZIE_HOME: /usr/lib/oozie
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_ZOOKEEPER_HOME: /usr/lib/zookeeper
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HUE_PLUGINS_HOME: /usr/lib/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: JSVC_HOME: /usr/libexec/bigtop-utils
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HDFS_HOME: /usr/lib/hadoop-hdfs
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_MR1_HOME: /usr/lib/hadoop-0.20-**
    mapreduce
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_MR2_HOME: /usr/lib/hadoop-mapreduce
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HADOOP_HOME: /usr/lib/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_IMPALA_HOME: /usr/lib/impala
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Logging to
    /var/log/cloudera-scm-agent/**cloudera-scm-agent.log
    Traceback (most recent call last):
    File "/usr/lib64/cmf/agent/src/cmf/**agent.py", line 2391, in
    <module>
    main()
    File "/usr/lib64/cmf/agent/src/cmf/**agent.py", line 2375, in main
    agent.start()
    File "/usr/lib64/cmf/agent/src/cmf/**agent.py", line 329, in start
    self.add_db_driver_jars()
    File "/usr/lib64/cmf/agent/src/cmf/**agent.py", line 1400, in
    add_db_driver_ja>>/usr/lib64/**cmf/agent/src/cmf/agent.py:31:
    DeprecationWarning: the sha module is deprecated; use the hashlib module
    instead
    import sha
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO SCM Agent
    Version: 4.6.0
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Using
    directory: /var/run/cloudera-scm-agent
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Using
    supervisor binary path: /usr/lib64/cmf/agent/src/cmf/.**
    ./../build/env/bin/**supervisord
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Adding env
    vars that start with CMF_AGENT_
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: HIVE_DEFAULT_XML: /etc/hive/conf.dist/hive-
    **default.xml
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: MGMT_HOME: /usr/share/cmf
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HTTPFS_HOME: /usr/lib/hadoop-httpfs
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_FLUME_HOME: /usr/lib/flume-ng
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HBASE_HOME: /usr/lib/hbase
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HUE_HOME: /usr/share/hue
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: TOMCAT_HOME: /usr/lib/bigtop-tomcat
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_PIG_HOME: /usr/lib/pig
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HIVE_HOME: /usr/lib/hive
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_YARN_HOME: /usr/lib/hadoop-yarn
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HADOOP_BIN: /usr/bin/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_OOZIE_HOME: /usr/lib/oozie
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_ZOOKEEPER_HOME: /usr/lib/zookeeper
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HUE_PLUGINS_HOME: /usr/lib/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: JSVC_HOME: /usr/libexec/bigtop-utils
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HDFS_HOME: /usr/lib/hadoop-hdfs
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_MR1_HOME: /usr/lib/hadoop-0.20-**
    mapreduce
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_MR2_HOME: /usr/lib/hadoop-mapreduce
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HADOOP_HOME: /usr/lib/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_IMPALA_HOME: /usr/lib/impala
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Logging to
    /var/log/cloudera-scm-agent/**cloudera-scm-agent.log
    Traceback (most recent call last):
    File "/usr/lib64/cmf/agent/src/cmf/**agent.py", line 2391, in
    <module>
    main()
    File "/usr/lib64/cmf/agent/src/cmf/**agent.py", line 2375, in main
    agent.start()
    File "/usr/lib64/cmf/agent/src/cmf/**agent.py", line 329, in start
    self.add_db_driver_jars()
    File "/usr/lib64/cmf/agent/src/cmf/**agent.py", line 1400, in
    add_db_driver_jars
    for jar in (mysql_jar, oracle_jar, postgresql_jar):
    UnboundLocalError: local variable 'postgresql_jar' referenced before
    assignment
    rs
    for jar in (mysql_jar, oracle_jar, postgresql_jar):
    UnboundLocalError: local variable 'postgresql_jar' referenced before
    assignment
    *END (0) *
    *BEGIN tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.log |
    sed 's/^/>>/' *
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Deleting
    process 1014-mapreduce-TASKTRACKER
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Triggering
    supervisord update.
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Retiring
    process 1014-mapreduce-TASKTRACKER
    [07/Jun/2013 22:59:32 +0000] 695 MonitorDaemon-Reporter
    throttling_logger ERROR Error sending messages to firehose:
    mgmt1-HOSTMONITOR-**d889e476015d035afbfed2002d76de**2d
    Traceback (most recent call last):
    File "/usr/lib64/cmf/agent/src/cmf/**monitor/firehose.py", line 67,
    in send
    self._requestor.request('**sendAgentMessages',
    dict(messages=messages))
    File "/usr/lib64/cmf/agent/build/**env/lib/python2.6/site-**
    packages/avro-1.6.3-py2.6.egg/**avro/ipc.py", line 145, in request
    return self.issue_request(call_**request, message_name,
    request_datum)
    File "/usr/lib64/cmf/agent/build/**env/lib/python2.6/site-**
    packages/avro-1.6.3-py2.6.egg/**avro/ipc.py", line 256, in
    issue_request
    call_response = self.transceiver.transceive(**call_request)
    File "/usr/lib64/cmf/agent/build/**env/lib/python2.6/site-**
    packages/avro-1.6.3-py2.6.egg/**avro/ipc.py", line 485, in transceive
    result = self.read_framed_message()
    File "/usr/lib64/cmf/agent/build/**env/lib/python2.6/site-**
    packages/avro-1.6.3-py2.6.egg/**avro/ipc.py", line 489, in
    read_framed_message
    response = self.conn.getresponse()
    File "/usr/lib64/python2.6/httplib.**py", line 990, in getresponse
    response.begin()
    File "/usr/lib64/python2.6/httplib.**py", line 391, in begin
    version, status, reason = self._read_status()
    File "/usr/lib64/python2.6/httplib.**py", line 355, in _read_status
    raise BadStatusLine(line)
    BadStatusLine
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Stopping
    agent...
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO No cgrouped
    processes running; unmounting any cgroup roots
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO No processes
    are being managed; Supervisor will shutdown.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Shutting down
    supervisord, pid 715
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Successfully
    shutdown supervisord, pid 715
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE HTTP Server cherrypy._cpwsgi_server.**
    CPWSGIServer(('dn6.lab10.**adconion.com <http://dn6.lab10.adconion.com>',
    9000)) shut down
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Stopped thread '_TimeoutMonitor'.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE HTTP Server cherrypy._cpwsgi_server.**
    CPWSGIServer(('dn6.lab10.**adconion.com <http://dn6.lab10.adconion.com>',
    9000)) already shut down
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE No thread running for None.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus EXITING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus EXITED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Agent exiting;
    caught signal 15
    [07/Jun/2013 23:02:30 +0000] 12620 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:02:30 +0000] 12620 Dummy-1 agent INFO Stopping
    agent...
    [07/Jun/2013 23:20:25 +0000] 14136 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:20:25 +0000] 14136 Dummy-1 agent INFO Stopping
    agent...
    [07/Jun/2013 23:51:20 +0000] 15580 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:51:20 +0000] 15580 Dummy-1 agent INFO Stopping
    agent...
    [07/Jun/2013 23:53:21 +0000] 16382 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:53:21 +0000] 16382 Dummy-1 agent INFO Stopping
    agent...
    [08/Jun/2013 00:05:13 +0000] 17261 MainThread agent INFO No command
    line vars
    [08/Jun/2013 00:05:13 +0000] 17261 Dummy-1 agent INFO Stopping
    agent...
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO No command
    line vars
    [08/Jun/2013 00:43:29 +0000] 20804 Dummy-1 agent INFO Stopping
    agent...
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Deleting
    process 1014-mapreduce-TASKTRACKER
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Triggering
    supervisord update.
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Retiring
    process 1014-mapreduce-TASKTRACKER
    [07/Jun/2013 22:59:32 +0000] 695 MonitorDaemon-Reporter
    throttling_logger ERROR Error sending messages to firehose:
    mgmt1-HOSTMONITOR-**d889e476015d035afbfed2002d76de**2d
    Traceback (most recent call last):
    File "/usr/lib64/cmf/agent/src/cmf/**monitor/firehose.py", line 67,
    in send
    self._requestor.request('**sendAgentMessages',
    dict(messages=messages))
    File "/usr/lib64/cmf/agent/build/**env/lib/python2.6/site-**
    packages/avro-1.6.3-py2.6.egg/**avro/ipc.py", line 145, in request
    return self.issue_request(call_**request, message_name,
    request_datum)
    File "/usr/lib64/cmf/agent/build/**env/lib/python2.6/site-**
    packages/avro-1.6.3-py2.6.egg/**avro/ipc.py", line 256, in
    issue_request
    call_response = self.transceiver.transceive(**call_request)
    File "/usr/lib64/cmf/agent/build/**env/lib/python2.6/site-**
    packages/avro-1.6.3-py2.6.egg/**avro/ipc.py", line 485, in transceive
    result = self.read_framed_message()
    File "/usr/lib64/cmf/agent/build/**env/lib/python2.6/site-**
    packages/avro-1.6.3-py2.6.egg/**avro/ipc.py", line 489, in
    read_framed_message
    response = self.conn.getresponse()
    File "/usr/lib64/python2.6/httplib.**py", line 990, in getresponse
    response.begin()
    File "/usr/lib64/python2.6/httplib.**py", line 391, in begin
    version, status, reason = self._read_status()
    File "/usr/lib64/python2.6/httplib.**py", line 355, in _read_status
    raise BadStatusLine(line)
    BadStatusLine
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Stopping
    agent...
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO No cgrouped
    processes running; unmounting any cgroup roots
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO No processes
    are being managed; Supervisor will shutdown.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Shutting down
    supervisord, pid 715
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Successfully
    shutdown supervisord, pid 715
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE HTTP Server cherrypy._cpwsgi_server.**
    CPWSGIServer(('dn6.lab10.**adconion.com <http://dn6.lab10.adconion.com>',
    9000)) shut down
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Stopped thread '_TimeoutMonitor'.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE HTTP Server cherrypy._cpwsgi_server.**
    CPWSGIServer(('dn6.lab10.**adconion.com <http://dn6.lab10.adconion.com>',
    9000)) already shut down
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE No thread running for None.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus EXITING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus EXITED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Agent exiting;
    caught signal 15
    [07/Jun/2013 23:02:30 +0000] 12620 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:02:30 +0000] 12620 Dummy-1 agent INFO Stopping
    agent...
    [07/Jun/2013 23:20:25 +0000] 14136 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:20:25 +0000] 14136 Dummy-1 agent INFO Stopping
    agent...
    [07/Jun/2013 23:51:20 +0000] 15580 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:51:20 +0000] 15580 Dummy-1 agent INFO Stopping
    agent...
    [07/Jun/2013 23:53:21 +0000] 16382 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:53:21 +0000] 16382 Dummy-1 agent INFO Stopping
    agent...
    [08/Jun/2013 00:05:13 +0000] 17261 MainThread agent INFO No command
    line vars
    [08/Jun/2013 00:05:13 +0000] 17261 Dummy-1 agent INFO Stopping
    agent...
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO No command
    line vars
    [08/Jun/2013 00:43:29 +0000] 20804 Dummy-1 agent INFO Stopping
    agent...
    *END (0) *
    end of agent logs.
    scm agent could not be started, giving up
    waiting for rollback request

    I hope you can help.

    Thanks,
    Ben
    On Friday, June 7, 2013 3:55:09 PM UTC-7, Benjamin Kim wrote:

    I just upgraded the CM server in EC2 to 4.6 from 4.5.2. I am trying to
    upgrade the node agents to the 4.6 using the wizard. I have 11 EC2 nodes in
    the cluster. When it gets to the Upgrade Hosts page, it cannot connect to
    any of the nodes in EC2. The names of all the nodes have been replaced with
    names like "i-4f792e33". I can see the original hosts with their
    correct names on the Hosts page. What can I do to fix this?

    Thanks,
    Ben

    --
    Thanks,
    Darren

    --
    Thanks,
    Darren
  • Benjamin Kim at Jun 10, 2013 at 8:52 pm ⇧
    Hi Darren,

    I switched everything over to using a symlink for
    /var/lib/cloudera-scm-server-db --> /u00/lib/cloudera-scm-server-db. It
    works without any errors or problems. Thanks for this suggestion. I forgot
    about symlinks.

    Cheers,
    Ben
    On Monday, June 10, 2013 10:30:02 AM UTC-7, Darren Lo wrote:

    Hi Ben,

    The embedded postgresql is intended to be a "quick-start" option and is
    not configurable. We assume that if someone wants to customize their
    database, then they will just set one up on their own.

    I expect that CM code won't correctly detect the embedded postgresql DB if
    it's in a different location. It could even error out. If the database is
    working fine, then go ahead and continue to use it, but treat it as a
    non-embedded postgresql. This means you should remove or rename
    /etc/cloudera-scm-server/db.mgmt.properties, and don't expect embedded db
    features such as automatic user creation for management services and Hive
    (though we'll still create the tables).

    If you really want to try and use the embedded db in a custom location,
    you might want to make a simlink from the normal
    /var/lib/cloudera-scm-server-db/data to your new dir. This will probably
    work, but I can't guarantee anything here.

    Thanks,
    Darren


    On Mon, Jun 10, 2013 at 9:16 AM, Benjamin Kim <bbui...@gmail.com<javascript:>
    wrote:
    Hi Darren,

    I did something more drastic. I uninstalled CM and reinstalled CM4.6 from
    scratch. After doing this, it worked. Since the root directories on my
    m1.large EC2 instances don't have much disk space, I had to change the
    Cloudera Manager log and embedded PostgreSQL DB file locations to a much
    larger secondary drive. I modified the scripts, cloudera-scm-server and
    cloudera-scm-server-db, in the /etc/init.d directory directly by changing
    references to the data directory from /var/lib to /u00/lib and references
    to the log directory from /var/log to /var/lib. Is this the right place to
    do these changes? I don't see any of these options being configurable in
    Cloudera Manager itself.

    On another note, I'll be troubleshooting the cluster to see why all the
    health checks are bad. But, all the services seem to be working fine.

    Thanks,
    Ben

    On Sunday, June 9, 2013 10:50:46 PM UTC-7, Darren Lo wrote:

    Hi Ben,

    Sorry for the toruble, and thanks for finding this bug! I've reproduced
    the error locally and I will make sure this is fixed in 4.6.1.

    In the meantime, you can get past this bug by removing the export line
    for CLOUDERA_POSTGRESQL_JDBC_JAR from /etc/default/cloudera-**scm-agent
    on all hosts. It isn't necessary anymore in 4.6. The agent code will
    correctly locate the jar in the cm lib dir by default (normally
    /usr/share/cmf/lib).

    Let me know if it works!

    Thanks,
    Darren

    On Fri, Jun 7, 2013 at 5:51 PM, Benjamin Kim wrote:

    I was able to fix the problem by going into the scm db and modifying
    the hosts table to have the identifier also be the name. But now, I'm
    having another issue. The CM Agents will not start up.

    Here is the log trace returned.

    /tmp/scm_prepare_node.**jChJxhLv
    using SSH_CLIENT to get the SCM hostname: 10.124.243.21 37046 22
    opening logging file descriptor
    Starting installation script... Acquiring installation lock...
    *BEGIN flock 4 *
    *END (0) *
    Detecting root privileges...
    effective UID is 0
    Detecting distribution...
    *BEGIN grep Tikanga /etc/redhat-release *
    *END (1) *
    *BEGIN grep 'CentOS release 5' /etc/redhat-release *
    *END (1) *
    *BEGIN grep 'Scientific Linux release 5' /etc/redhat-release *
    *END (1) *
    *BEGIN grep Santiago /etc/redhat-release *
    *END (1) *
    *BEGIN grep 'CentOS Linux release 6' /etc/redhat-release *
    *END (1) *
    *BEGIN grep 'CentOS release 6' /etc/redhat-release *
    CentOS release 6.3 (Final)
    *END (0) *
    /etc/redhat-release ==> CentOS 6
    Detecting Cloudera Manager Server...
    *BEGIN host -t PTR 10.124.243.21 *
    21.243.124.10.in-addr.arpa domain name pointer
    ip-10-124-243-21.ec2.internal.**
    *END (0) *
    using ip-10-124-243-21.ec2.internal as scm server hostname
    *BEGIN which python *
    /usr/bin/python
    *END (0) *
    *BEGIN python -c 'import socket; import sys; s =
    socket.socket(socket.AF_INET); s.settimeout(5.0); s.connect((sys.argv[1],
    int(sys.argv[2]))); s.close();' ip-10-124-243-21.ec2.internal 7182 *
    *END (0) *
    *BEGIN which wget *
    /usr/bin/wget
    *END (0) *
    *BEGIN wget -qO- -T 1 -t 1 http://169.254.169.254/latest/
    meta-data/public-hostname && /bin/echo *
    ec2-107-21-133-38.compute-1.**amazonaws.com<http://ec2-107-21-133-38.compute-1.amazonaws.com>

    *END (0) *
    *BEGIN wget -qO- -T 1 -t 1 http://169.254.169.254/latest/
    meta-data/instance-id && /bin/echo *
    i-05653279
    *END (0) *
    Performing cloud-specific configuration...
    *BEGIN /sbin/chkconfig iptables *
    *END (1) *
    *BEGIN /sbin/service iptables status | grep Chain *
    Chain INPUT (policy ACCEPT)
    Chain FORWARD (policy ACCEPT)
    Chain OUTPUT (policy ACCEPT)
    Chain RH-Firewall-1-INPUT (2 references)
    *END (0) *
    *BEGIN /sbin/service iptables stop *
    iptables: Flushing firewall rules: [60G[ [0;32m OK [0;39m]
    iptables: Setting chains to policy ACCEPT: filter [60G[ [0;32m OK
    [0;39m]
    iptables: Unloading modules: [60G[ [0;32m OK [0;39m]
    *END (0) *
    *BEGIN getenforce *
    Disabled
    *END (0) *
    *BEGIN grep -e ^SELINUX=enforcing /etc/selinux/config *
    *END (1) *
    *BEGIN df -h | grep -o -E "^/dev/[^[:space:]]*" *
    /dev/xvde1
    /dev/md0
    *END (0) *
    *BEGIN awk 'FNR > 2 {print $NF}' /proc/partitions *
    xvdf
    xvdf1
    xvdf2
    xvdg
    xvdg1
    xvdg2
    xvde
    xvde1
    md0
    *END (0) *
    Installing package repositories...
    validating format of repository file /tmp/scm_prepare_node.**
    jChJxhLv/repos/rhel6/cloudera-**manager.repo
    installing repository file /tmp/scm_prepare_node.**
    jChJxhLv/repos/rhel6/cloudera-**manager.repo
    repository file /tmp/scm_prepare_node.**jChJxhLv/repos/rhel6/cloudera-*
    *manager.repo installed
    Refreshing package metadata...
    *BEGIN yum clean all *
    Loaded plugins: fastestmirror, presto
    Cleaning repos: base cloudera-cdh4 cloudera-impala cloudera-manager
    epel extras
    : updates
    Cleaning up Everything
    0 delta-package files removed, by presto
    *END (0) *
    *BEGIN rm -Rf /var/cache/yum/x86_64 *
    *END (0) *
    *BEGIN yum makecache *
    Loaded plugins: fastestmirror, presto
    Metadata Cache Created
    *END (0) *
    Installing jdk package...
    *BEGIN yum list installed jdk *
    Loaded plugins: fastestmirror, presto
    Installed Packages
    jdk.x86_64 2000:1.6.0_31-fcs @cloudera-manager
    *END (0) *
    *BEGIN echo jdk cloudera-manager-agent cloudera-manager-daemons | grep
    jdk *
    jdk cloudera-manager-agent cloudera-manager-daemons
    *END (0) *
    *BEGIN yum info jdk *
    Loaded plugins: fastestmirror, presto
    Determining fastest mirrors
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.**fedoraproject.org<http://s3-mirror-us-east-1.fedoraproject.org>

    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Installed Packages
    Name : jdk
    Arch : x86_64
    Epoch : 2000
    Version : 1.6.0_31
    Release : fcs
    Size : 143 M
    Repo : installed
    From repo : cloudera-manager
    Summary : Java(TM) Platform Standard Edition Development Kit
    URL : http://java.sun.com/
    License : Copyright (c) 2011, Oracle and/or its affiliates. All rights
    : reserved. Also under other license(s) as shown at the Description
    : field.
    Description : The Java Platform Standard Edition Development Kit (JDK)
    includes
    : both the runtime environment (Java virtual machine, the Java
    : platform classes and supporting files) and development tools
    : (compilers, debuggers, tool libraries and other tools).
    :
    : The JDK is a development environment for building applications,
    : applets and components that can be deployed with the Java Platform
    : Standard Edition Runtime Environment.

    *END (0) *
    *BEGIN yum -y install jdk.x86_64 *
    Loaded plugins: fastestmirror, presto
    Loading mirror speeds from cached hostfile
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.**fedoraproject.org<http://s3-mirror-us-east-1.fedoraproject.org>

    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Setting up Install Process
    Package 2000:jdk-1.6.0_31-fcs.x86_64 already installed and latest
    version
    Nothing to do
    *END (0) *
    remote package jdk installed
    Installing cloudera-manager-agent package...
    *BEGIN yum list installed cloudera-manager-agent *
    Loaded plugins: fastestmirror, presto
    Installed Packages
    cloudera-manager-agent.x86_64 4.6.0-1.cm460.p0.140 @cloudera-manager
    *END (0) *
    *BEGIN echo jdk cloudera-manager-agent cloudera-manager-daemons | grep
    cloudera-manager-agent *
    *END (0) *
    jdk cloudera-manager-agent cloudera-manager-daemons
    *BEGIN yum info cloudera-manager-agent *
    Loaded plugins: fastestmirror, presto
    Loading mirror speeds from cached hostfile
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.**fedoraproject.org<http://s3-mirror-us-east-1.fedoraproject.org>

    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Installed Packages
    Name : cloudera-manager-agent
    Arch : x86_64
    Version : 4.6.0
    Release : 1.cm460.p0.140
    Size : 25 M
    Repo : installed
    From repo : cloudera-manager
    Summary : The Cloudera Manager Agent
    URL : http://www.cloudera.com
    License : Proprietary
    Description : The Cloudera Manager Agent.
    :
    : The Agent is deployed to machines running services managed by
    : Cloudera Manager.

    *END (0) *
    Version : 4.6.0
    Release : 1.cm460.p0.140
    *BEGIN yum -y install cloudera-manager-agent *
    Loaded plugins: fastestmirror, presto
    Loading mirror speeds from cached hostfile
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.**fedoraproject.org<http://s3-mirror-us-east-1.fedoraproject.org>

    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Setting up Install Process
    Package cloudera-manager-agent-4.6.0-**1.cm460.p0.140.x86_64 already
    installed and latest version
    Nothing to do
    *END (0) *
    remote package cloudera-manager-agent installed
    Installing cloudera-manager-daemons package...
    *BEGIN yum list installed cloudera-manager-daemons *
    Loaded plugins: fastestmirror, presto
    Installed Packages
    cloudera-manager-daemons.x86_**64 4.6.0-1.cm460.p0.140
    @cloudera-manager
    *END (0) *
    *BEGIN echo jdk cloudera-manager-agent cloudera-manager-daemons | grep
    cloudera-manager-daemons *
    jdk cloudera-manager-agent cloudera-manager-daemons
    *END (0) *
    *BEGIN yum info cloudera-manager-daemons *
    Loaded plugins: fastestmirror, presto
    Loading mirror speeds from cached hostfile
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.**fedoraproject.org<http://s3-mirror-us-east-1.fedoraproject.org>

    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Installed Packages
    Name : cloudera-manager-daemons
    Arch : x86_64
    Version : 4.6.0
    Release : 1.cm460.p0.140
    Size : 175 M
    Repo : installed
    From repo : cloudera-manager
    Summary : Provides daemons for monitoring Hadoop and related tools.
    URL : http://www.cloudera.com
    License : Proprietary
    Description : This package includes daemons for monitoring and managing
    Hadoop.

    *END (0) *
    Version : 4.6.0
    Release : 1.cm460.p0.140
    *BEGIN yum -y install cloudera-manager-daemons *
    Loaded plugins: fastestmirror, presto
    Loading mirror speeds from cached hostfile
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.**fedoraproject.org<http://s3-mirror-us-east-1.fedoraproject.org>

    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Setting up Install Process
    Package cloudera-manager-daemons-4.6.**0-1.cm460.p0.140.x86_64 already
    installed and latest version
    Nothing to do
    *END (0) *
    remote package cloudera-manager-daemons installed
    Configuring Cloudera Manager Agent...
    *BEGIN sed -e 's/\(CMF_AGENT_ARGS=\).*/\1"--host_id i-05653279"/' -i
    /etc/default/cloudera-scm-agent *
    *END (0) *
    *BEGIN grep server_host=ip-10-124-243-21.ec2.internal
    /etc/cloudera-scm-agent/config.ini *
    server_host=ip-10-124-243-21.**ec2.internal
    *END (0) *
    scm agent is already configured
    Starting Cloudera Manager Agent...
    *BEGIN /sbin/service cloudera-scm-agent status | grep running *
    *END (1) *
    *BEGIN /sbin/service cloudera-scm-agent start *
    Starting cloudera-scm-agent: [60G[ [0;31mFAILED [0;39m]
    *END (1) *
    agent logs:
    *BEGIN tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.out
    sed 's/^/>>/' *
    /usr/lib64/cmf/agent/src/**cmf/agent.py:31: DeprecationWarning: the
    sha module is deprecated; use the hashlib module instead
    import sha
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO SCM Agent
    Version: 4.6.0
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Using
    directory: /var/run/cloudera-scm-agent
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Using
    supervisor binary path: /usr/lib64/cmf/agent/src/cmf/.**
    ./../build/env/bin/**supervisord
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Adding env
    vars that start with CMF_AGENT_
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: HIVE_DEFAULT_XML: /etc/hive/conf.dist/hive-
    **default.xml
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: MGMT_HOME: /usr/share/cmf
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HTTPFS_HOME: /usr/lib/hadoop-httpfs
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_FLUME_HOME: /usr/lib/flume-ng
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HBASE_HOME: /usr/lib/hbase
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HUE_HOME: /usr/share/hue
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: TOMCAT_HOME: /usr/lib/bigtop-tomcat
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_PIG_HOME: /usr/lib/pig
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HIVE_HOME: /usr/lib/hive
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_YARN_HOME: /usr/lib/hadoop-yarn
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HADOOP_BIN: /usr/bin/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_OOZIE_HOME: /usr/lib/oozie
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_ZOOKEEPER_HOME: /usr/lib/zookeeper
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HUE_PLUGINS_HOME: /usr/lib/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: JSVC_HOME: /usr/libexec/bigtop-utils
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HDFS_HOME: /usr/lib/hadoop-hdfs
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_MR1_HOME: /usr/lib/hadoop-0.20-**
    mapreduce
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_MR2_HOME: /usr/lib/hadoop-mapreduce
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HADOOP_HOME: /usr/lib/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_IMPALA_HOME: /usr/lib/impala
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Logging to
    /var/log/cloudera-scm-agent/**cloudera-scm-agent.log
    Traceback (most recent call last):
    File "/usr/lib64/cmf/agent/src/cmf/**agent.py", line 2391, in
    <module>
    main()
    File "/usr/lib64/cmf/agent/src/cmf/**agent.py", line 2375, in main
    agent.start()
    File "/usr/lib64/cmf/agent/src/cmf/**agent.py", line 329, in start
    self.add_db_driver_jars()
    File "/usr/lib64/cmf/agent/src/cmf/**agent.py", line 1400, in
    add_db_driver_ja>>/usr/lib64/**cmf/agent/src/cmf/agent.py:31:
    DeprecationWarning: the sha module is deprecated; use the hashlib module
    instead
    import sha
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO SCM Agent
    Version: 4.6.0
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Using
    directory: /var/run/cloudera-scm-agent
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Using
    supervisor binary path: /usr/lib64/cmf/agent/src/cmf/.**
    ./../build/env/bin/**supervisord
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Adding env
    vars that start with CMF_AGENT_
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: HIVE_DEFAULT_XML: /etc/hive/conf.dist/hive-
    **default.xml
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: MGMT_HOME: /usr/share/cmf
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HTTPFS_HOME: /usr/lib/hadoop-httpfs
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_FLUME_HOME: /usr/lib/flume-ng
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HBASE_HOME: /usr/lib/hbase
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HUE_HOME: /usr/share/hue
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: TOMCAT_HOME: /usr/lib/bigtop-tomcat
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_PIG_HOME: /usr/lib/pig
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HIVE_HOME: /usr/lib/hive
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_YARN_HOME: /usr/lib/hadoop-yarn
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HADOOP_BIN: /usr/bin/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_OOZIE_HOME: /usr/lib/oozie
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_ZOOKEEPER_HOME: /usr/lib/zookeeper
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HUE_PLUGINS_HOME: /usr/lib/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: JSVC_HOME: /usr/libexec/bigtop-utils
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HDFS_HOME: /usr/lib/hadoop-hdfs
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_MR1_HOME: /usr/lib/hadoop-0.20-**
    mapreduce
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_MR2_HOME: /usr/lib/hadoop-mapreduce
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HADOOP_HOME: /usr/lib/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_IMPALA_HOME: /usr/lib/impala
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Logging to
    /var/log/cloudera-scm-agent/**cloudera-scm-agent.log
    Traceback (most recent call last):
    File "/usr/lib64/cmf/agent/src/cmf/**agent.py", line 2391, in
    <module>
    main()
    File "/usr/lib64/cmf/agent/src/cmf/**agent.py", line 2375, in main
    agent.start()
    File "/usr/lib64/cmf/agent/src/cmf/**agent.py", line 329, in start
    self.add_db_driver_jars()
    File "/usr/lib64/cmf/agent/src/cmf/**agent.py", line 1400, in
    add_db_driver_jars
    for jar in (mysql_jar, oracle_jar, postgresql_jar):
    UnboundLocalError: local variable 'postgresql_jar' referenced before
    assignment
    rs
    for jar in (mysql_jar, oracle_jar, postgresql_jar):
    UnboundLocalError: local variable 'postgresql_jar' referenced before
    assignment
    *END (0) *
    *BEGIN tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.log
    sed 's/^/>>/' *
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Deleting
    process 1014-mapreduce-TASKTRACKER
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Triggering
    supervisord update.
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Retiring
    process 1014-mapreduce-TASKTRACKER
    [07/Jun/2013 22:59:32 +0000] 695 MonitorDaemon-Reporter
    throttling_logger ERROR Error sending messages to firehose:
    mgmt1-HOSTMONITOR-**d889e476015d035afbfed2002d76de**2d
    Traceback (most recent call last):
    File "/usr/lib64/cmf/agent/src/cmf/**monitor/firehose.py", line 67,
    in send
    self._requestor.request('**sendAgentMessages',
    dict(messages=messages))
    File "/usr/lib64/cmf/agent/build/**env/lib/python2.6/site-**
    packages/avro-1.6.3-py2.6.egg/**avro/ipc.py", line 145, in request
    return self.issue_request(call_**request, message_name,
    request_datum)
    File "/usr/lib64/cmf/agent/build/**env/lib/python2.6/site-**
    packages/avro-1.6.3-py2.6.egg/**avro/ipc.py", line 256, in
    issue_request
    call_response = self.transceiver.transceive(**call_request)
    File "/usr/lib64/cmf/agent/build/**env/lib/python2.6/site-**
    packages/avro-1.6.3-py2.6.egg/**avro/ipc.py", line 485, in transceive
    result = self.read_framed_message()
    File "/usr/lib64/cmf/agent/build/**env/lib/python2.6/site-**
    packages/avro-1.6.3-py2.6.egg/**avro/ipc.py", line 489, in
    read_framed_message
    response = self.conn.getresponse()
    File "/usr/lib64/python2.6/httplib.**py", line 990, in getresponse
    response.begin()
    File "/usr/lib64/python2.6/httplib.**py", line 391, in begin
    version, status, reason = self._read_status()
    File "/usr/lib64/python2.6/httplib.**py", line 355, in _read_status
    raise BadStatusLine(line)
    BadStatusLine
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Stopping
    agent...
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO No cgrouped
    processes running; unmounting any cgroup roots
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO No processes
    are being managed; Supervisor will shutdown.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Shutting down
    supervisord, pid 715
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Successfully
    shutdown supervisord, pid 715
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE HTTP Server cherrypy._cpwsgi_server.**
    CPWSGIServer(('dn6.lab10.**adconion.com <http://dn6.lab10.adconion.com>',
    9000)) shut down
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Stopped thread '_TimeoutMonitor'.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE HTTP Server cherrypy._cpwsgi_server.**
    CPWSGIServer(('dn6.lab10.**adconion.com <http://dn6.lab10.adconion.com>',
    9000)) already shut down
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE No thread running for None.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus EXITING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus EXITED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Agent exiting;
    caught signal 15
    [07/Jun/2013 23:02:30 +0000] 12620 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:02:30 +0000] 12620 Dummy-1 agent INFO Stopping
    agent...
    [07/Jun/2013 23:20:25 +0000] 14136 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:20:25 +0000] 14136 Dummy-1 agent INFO Stopping
    agent...
    [07/Jun/2013 23:51:20 +0000] 15580 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:51:20 +0000] 15580 Dummy-1 agent INFO Stopping
    agent...
    [07/Jun/2013 23:53:21 +0000] 16382 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:53:21 +0000] 16382 Dummy-1 agent INFO Stopping
    agent...
    [08/Jun/2013 00:05:13 +0000] 17261 MainThread agent INFO No command
    line vars
    [08/Jun/2013 00:05:13 +0000] 17261 Dummy-1 agent INFO Stopping
    agent...
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO No command
    line vars
    [08/Jun/2013 00:43:29 +0000] 20804 Dummy-1 agent INFO Stopping
    agent...
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Deleting
    process 1014-mapreduce-TASKTRACKER
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Triggering
    supervisord update.
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Retiring
    process 1014-mapreduce-TASKTRACKER
    [07/Jun/2013 22:59:32 +0000] 695 MonitorDaemon-Reporter
    throttling_logger ERROR Error sending messages to firehose:
    mgmt1-HOSTMONITOR-**d889e476015d035afbfed2002d76de**2d
    Traceback (most recent call last):
    File "/usr/lib64/cmf/agent/src/cmf/**monitor/firehose.py", line 67,
    in send
    self._requestor.request('**sendAgentMessages',
    dict(messages=messages))
    File "/usr/lib64/cmf/agent/build/**env/lib/python2.6/site-**
    packages/avro-1.6.3-py2.6.egg/**avro/ipc.py", line 145, in request
    return self.issue_request(call_**request, message_name,
    request_datum)
    File "/usr/lib64/cmf/agent/build/**env/lib/python2.6/site-**
    packages/avro-1.6.3-py2.6.egg/**avro/ipc.py", line 256, in
    issue_request
    call_response = self.transceiver.transceive(**call_request)
    File "/usr/lib64/cmf/agent/build/**env/lib/python2.6/site-**
    packages/avro-1.6.3-py2.6.egg/**avro/ipc.py", line 485, in transceive
    result = self.read_framed_message()
    File "/usr/lib64/cmf/agent/build/**env/lib/python2.6/site-**
    packages/avro-1.6.3-py2.6.egg/**avro/ipc.py", line 489, in
    read_framed_message
    response = self.conn.getresponse()
    File "/usr/lib64/python2.6/httplib.**py", line 990, in getresponse
    response.begin()
    File "/usr/lib64/python2.6/httplib.**py", line 391, in begin
    version, status, reason = self._read_status()
    File "/usr/lib64/python2.6/httplib.**py", line 355, in _read_status
    raise BadStatusLine(line)
    BadStatusLine
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Stopping
    agent...
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO No cgrouped
    processes running; unmounting any cgroup roots
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO No processes
    are being managed; Supervisor will shutdown.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Shutting down
    supervisord, pid 715
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Successfully
    shutdown supervisord, pid 715
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE HTTP Server cherrypy._cpwsgi_server.**
    CPWSGIServer(('dn6.lab10.**adconion.com <http://dn6.lab10.adconion.com>',
    9000)) shut down
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Stopped thread '_TimeoutMonitor'.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE HTTP Server cherrypy._cpwsgi_server.**
    CPWSGIServer(('dn6.lab10.**adconion.com <http://dn6.lab10.adconion.com>',
    9000)) already shut down
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE No thread running for None.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus EXITING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus EXITED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Agent exiting;
    caught signal 15
    [07/Jun/2013 23:02:30 +0000] 12620 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:02:30 +0000] 12620 Dummy-1 agent INFO Stopping
    agent...
    [07/Jun/2013 23:20:25 +0000] 14136 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:20:25 +0000] 14136 Dummy-1 agent INFO Stopping
    agent...
    [07/Jun/2013 23:51:20 +0000] 15580 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:51:20 +0000] 15580 Dummy-1 agent INFO Stopping
    agent...
    [07/Jun/2013 23:53:21 +0000] 16382 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:53:21 +0000] 16382 Dummy-1 agent INFO Stopping
    agent...
    [08/Jun/2013 00:05:13 +0000] 17261 MainThread agent INFO No command
    line vars
    [08/Jun/2013 00:05:13 +0000] 17261 Dummy-1 agent INFO Stopping
    agent...
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO No command
    line vars
    [08/Jun/2013 00:43:29 +0000] 20804 Dummy-1 agent INFO Stopping
    agent...
    *END (0) *
    end of agent logs.
    scm agent could not be started, giving up
    waiting for rollback request

    I hope you can help.

    Thanks,
    Ben
    On Friday, June 7, 2013 3:55:09 PM UTC-7, Benjamin Kim wrote:

    I just upgraded the CM server in EC2 to 4.6 from 4.5.2. I am trying to
    upgrade the node agents to the 4.6 using the wizard. I have 11 EC2 nodes in
    the cluster. When it gets to the Upgrade Hosts page, it cannot connect to
    any of the nodes in EC2. The names of all the nodes have been replaced with
    names like "i-4f792e33". I can see the original hosts with their
    correct names on the Hosts page. What can I do to fix this?

    Thanks,
    Ben

    --
    Thanks,
    Darren

    --
    Thanks,
    Darren
  • Darren Lo at Jun 10, 2013 at 8:57 pm ⇧
    Glad to hear it's working!

    On Mon, Jun 10, 2013 at 1:52 PM, Benjamin Kim wrote:

    Hi Darren,

    I switched everything over to using a symlink for
    /var/lib/cloudera-scm-server-db --> /u00/lib/cloudera-scm-server-db. It
    works without any errors or problems. Thanks for this suggestion. I forgot
    about symlinks.

    Cheers,
    Ben

    On Monday, June 10, 2013 10:30:02 AM UTC-7, Darren Lo wrote:

    Hi Ben,

    The embedded postgresql is intended to be a "quick-start" option and is
    not configurable. We assume that if someone wants to customize their
    database, then they will just set one up on their own.

    I expect that CM code won't correctly detect the embedded postgresql DB
    if it's in a different location. It could even error out. If the database
    is working fine, then go ahead and continue to use it, but treat it as a
    non-embedded postgresql. This means you should remove or rename
    /etc/cloudera-scm-server/db.**mgmt.properties, and don't expect embedded
    db features such as automatic user creation for management services and
    Hive (though we'll still create the tables).

    If you really want to try and use the embedded db in a custom location,
    you might want to make a simlink from the normal
    /var/lib/cloudera-scm-server-**db/data to your new dir. This will
    probably work, but I can't guarantee anything here.

    Thanks,
    Darren

    On Mon, Jun 10, 2013 at 9:16 AM, Benjamin Kim wrote:

    Hi Darren,

    I did something more drastic. I uninstalled CM and reinstalled CM4.6
    from scratch. After doing this, it worked. Since the root directories on my
    m1.large EC2 instances don't have much disk space, I had to change the
    Cloudera Manager log and embedded PostgreSQL DB file locations to a much
    larger secondary drive. I modified the scripts, cloudera-scm-server and
    cloudera-scm-server-db, in the /etc/init.d directory directly by changing
    references to the data directory from /var/lib to /u00/lib and references
    to the log directory from /var/log to /var/lib. Is this the right place to
    do these changes? I don't see any of these options being configurable in
    Cloudera Manager itself.

    On another note, I'll be troubleshooting the cluster to see why all the
    health checks are bad. But, all the services seem to be working fine.

    Thanks,
    Ben

    On Sunday, June 9, 2013 10:50:46 PM UTC-7, Darren Lo wrote:

    Hi Ben,

    Sorry for the toruble, and thanks for finding this bug! I've reproduced
    the error locally and I will make sure this is fixed in 4.6.1.

    In the meantime, you can get past this bug by removing the export line
    for CLOUDERA_POSTGRESQL_JDBC_JAR from /etc/default/cloudera-**scm**-agent
    on all hosts. It isn't necessary anymore in 4.6. The agent code will
    correctly locate the jar in the cm lib dir by default (normally
    /usr/share/cmf/lib).

    Let me know if it works!

    Thanks,
    Darren

    On Fri, Jun 7, 2013 at 5:51 PM, Benjamin Kim wrote:

    I was able to fix the problem by going into the scm db and modifying
    the hosts table to have the identifier also be the name. But now, I'm
    having another issue. The CM Agents will not start up.

    Here is the log trace returned.

    /tmp/scm_prepare_node.**jChJxhLv**
    using SSH_CLIENT to get the SCM hostname: 10.124.243.21 37046 22
    opening logging file descriptor
    Starting installation script... Acquiring installation lock...
    *BEGIN flock 4 *
    *END (0) *
    Detecting root privileges...
    effective UID is 0
    Detecting distribution...
    *BEGIN grep Tikanga /etc/redhat-release *
    *END (1) *
    *BEGIN grep 'CentOS release 5' /etc/redhat-release *
    *END (1) *
    *BEGIN grep 'Scientific Linux release 5' /etc/redhat-release *
    *END (1) *
    *BEGIN grep Santiago /etc/redhat-release *
    *END (1) *
    *BEGIN grep 'CentOS Linux release 6' /etc/redhat-release *
    *END (1) *
    *BEGIN grep 'CentOS release 6' /etc/redhat-release *
    CentOS release 6.3 (Final)
    *END (0) *
    /etc/redhat-release ==> CentOS 6
    Detecting Cloudera Manager Server...
    *BEGIN host -t PTR 10.124.243.21 *
    21.243.124.10.in-addr.arpa domain name pointer
    ip-10-124-243-21.ec2.internal.****
    *END (0) *
    using ip-10-124-243-21.ec2.internal as scm server hostname
    *BEGIN which python *
    /usr/bin/python
    *END (0) *
    *BEGIN python -c 'import socket; import sys; s =
    socket.socket(socket.AF_INET); s.settimeout(5.0); s.connect((sys.argv[1],
    int(sys.argv[2]))); s.close();' ip-10-124-243-21.ec2.internal 7182 *
    *END (0) *
    *BEGIN which wget *
    /usr/bin/wget
    *END (0) *
    *BEGIN wget -qO- -T 1 -t 1 http://169.254.169.254/latest/
    meta-data/public-hostname && /bin/echo *
    ec2-107-21-133-38.compute-1.**am**azonaws.com<http://ec2-107-21-133-38.compute-1.amazonaws.com>

    *END (0) *
    *BEGIN wget -qO- -T 1 -t 1 http://169.254.169.254/latest/
    meta-data/instance-id && /bin/echo *
    i-05653279
    *END (0) *
    Performing cloud-specific configuration...
    *BEGIN /sbin/chkconfig iptables *
    *END (1) *
    *BEGIN /sbin/service iptables status | grep Chain *
    Chain INPUT (policy ACCEPT)
    Chain FORWARD (policy ACCEPT)
    Chain OUTPUT (policy ACCEPT)
    Chain RH-Firewall-1-INPUT (2 references)
    *END (0) *
    *BEGIN /sbin/service iptables stop *
    iptables: Flushing firewall rules: [60G[ [0;32m OK [0;39m]
    iptables: Setting chains to policy ACCEPT: filter [60G[ [0;32m OK
    [0;39m]
    iptables: Unloading modules: [60G[ [0;32m OK [0;39m]
    *END (0) *
    *BEGIN getenforce *
    Disabled
    *END (0) *
    *BEGIN grep -e ^SELINUX=enforcing /etc/selinux/config *
    *END (1) *
    *BEGIN df -h | grep -o -E "^/dev/[^[:space:]]*" *
    /dev/xvde1
    /dev/md0
    *END (0) *
    *BEGIN awk 'FNR > 2 {print $NF}' /proc/partitions *
    xvdf
    xvdf1
    xvdf2
    xvdg
    xvdg1
    xvdg2
    xvde
    xvde1
    md0
    *END (0) *
    Installing package repositories...
    validating format of repository file /tmp/scm_prepare_node.**jChJxhLv*
    */repos/rhel6/cloudera-**manager.**repo
    installing repository file /tmp/scm_prepare_node.**jChJxhLv**
    /repos/rhel6/cloudera-**manager.**repo
    repository file /tmp/scm_prepare_node.**jChJxhLv**
    /repos/rhel6/cloudera-**manager.**repo installed
    Refreshing package metadata...
    *BEGIN yum clean all *
    Loaded plugins: fastestmirror, presto
    Cleaning repos: base cloudera-cdh4 cloudera-impala cloudera-manager
    epel extras
    : updates
    Cleaning up Everything
    0 delta-package files removed, by presto
    *END (0) *
    *BEGIN rm -Rf /var/cache/yum/x86_64 *
    *END (0) *
    *BEGIN yum makecache *
    Loaded plugins: fastestmirror, presto
    Metadata Cache Created
    *END (0) *
    Installing jdk package...
    *BEGIN yum list installed jdk *
    Loaded plugins: fastestmirror, presto
    Installed Packages
    jdk.x86_64 2000:1.6.0_31-fcs @cloudera-manager
    *END (0) *
    *BEGIN echo jdk cloudera-manager-agent cloudera-manager-daemons |
    grep jdk *
    jdk cloudera-manager-agent cloudera-manager-daemons
    *END (0) *
    *BEGIN yum info jdk *
    Loaded plugins: fastestmirror, presto
    Determining fastest mirrors
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.**fedoraproj**ect.org<http://s3-mirror-us-east-1.fedoraproject.org>

    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Installed Packages
    Name : jdk
    Arch : x86_64
    Epoch : 2000
    Version : 1.6.0_31
    Release : fcs
    Size : 143 M
    Repo : installed
    From repo : cloudera-manager
    Summary : Java(TM) Platform Standard Edition Development Kit
    URL : http://java.sun.com/
    License : Copyright (c) 2011, Oracle and/or its affiliates. All rights
    : reserved. Also under other license(s) as shown at the Description
    : field.
    Description : The Java Platform Standard Edition Development Kit (JDK)
    includes
    : both the runtime environment (Java virtual machine, the Java
    : platform classes and supporting files) and development tools
    : (compilers, debuggers, tool libraries and other tools).
    :
    : The JDK is a development environment for building applications,
    : applets and components that can be deployed with the Java Platform
    : Standard Edition Runtime Environment.

    *END (0) *
    *BEGIN yum -y install jdk.x86_64 *
    Loaded plugins: fastestmirror, presto
    Loading mirror speeds from cached hostfile
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.**fedoraproj**ect.org<http://s3-mirror-us-east-1.fedoraproject.org>

    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Setting up Install Process
    Package 2000:jdk-1.6.0_31-fcs.x86_64 already installed and latest
    version
    Nothing to do
    *END (0) *
    remote package jdk installed
    Installing cloudera-manager-agent package...
    *BEGIN yum list installed cloudera-manager-agent *
    Loaded plugins: fastestmirror, presto
    Installed Packages
    cloudera-manager-agent.x86_64 4.6.0-1.cm460.p0.140 @cloudera-manager
    *END (0) *
    *BEGIN echo jdk cloudera-manager-agent cloudera-manager-daemons |
    grep cloudera-manager-agent *
    *END (0) *
    jdk cloudera-manager-agent cloudera-manager-daemons
    *BEGIN yum info cloudera-manager-agent *
    Loaded plugins: fastestmirror, presto
    Loading mirror speeds from cached hostfile
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.**fedoraproj**ect.org<http://s3-mirror-us-east-1.fedoraproject.org>

    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Installed Packages
    Name : cloudera-manager-agent
    Arch : x86_64
    Version : 4.6.0
    Release : 1.cm460.p0.140
    Size : 25 M
    Repo : installed
    From repo : cloudera-manager
    Summary : The Cloudera Manager Agent
    URL : http://www.cloudera.com
    License : Proprietary
    Description : The Cloudera Manager Agent.
    :
    : The Agent is deployed to machines running services managed by
    : Cloudera Manager.

    *END (0) *
    Version : 4.6.0
    Release : 1.cm460.p0.140
    *BEGIN yum -y install cloudera-manager-agent *
    Loaded plugins: fastestmirror, presto
    Loading mirror speeds from cached hostfile
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.**fedoraproj**ect.org<http://s3-mirror-us-east-1.fedoraproject.org>

    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Setting up Install Process
    Package cloudera-manager-agent-4.6.0-**1**.cm460.p0.140.x86_64
    already installed and latest version
    Nothing to do
    *END (0) *
    remote package cloudera-manager-agent installed
    Installing cloudera-manager-daemons package...
    *BEGIN yum list installed cloudera-manager-daemons *
    Loaded plugins: fastestmirror, presto
    Installed Packages
    cloudera-manager-daemons.x86_**6**4 4.6.0-1.cm460.p0.140
    @cloudera-manager
    *END (0) *
    *BEGIN echo jdk cloudera-manager-agent cloudera-manager-daemons |
    grep cloudera-manager-daemons *
    jdk cloudera-manager-agent cloudera-manager-daemons
    *END (0) *
    *BEGIN yum info cloudera-manager-daemons *
    Loaded plugins: fastestmirror, presto
    Loading mirror speeds from cached hostfile
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.**fedoraproj**ect.org<http://s3-mirror-us-east-1.fedoraproject.org>

    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Installed Packages
    Name : cloudera-manager-daemons
    Arch : x86_64
    Version : 4.6.0
    Release : 1.cm460.p0.140
    Size : 175 M
    Repo : installed
    From repo : cloudera-manager
    Summary : Provides daemons for monitoring Hadoop and related tools.
    URL : http://www.cloudera.com
    License : Proprietary
    Description : This package includes daemons for monitoring and
    managing Hadoop.

    *END (0) *
    Version : 4.6.0
    Release : 1.cm460.p0.140
    *BEGIN yum -y install cloudera-manager-daemons *
    Loaded plugins: fastestmirror, presto
    Loading mirror speeds from cached hostfile
    * base: mirrors.advancedhosters.com
    * epel: s3-mirror-us-east-1.**fedoraproj**ect.org<http://s3-mirror-us-east-1.fedoraproject.org>

    * extras: mirror.cogentco.com
    * updates: mirrors.lga7.us.voxel.net
    Setting up Install Process
    Package cloudera-manager-daemons-4.6.**0**-1.cm460.p0.140.x86_64
    already installed and latest version
    Nothing to do
    *END (0) *
    remote package cloudera-manager-daemons installed
    Configuring Cloudera Manager Agent...
    *BEGIN sed -e 's/\(CMF_AGENT_ARGS=\).*/\1"--host_id i-05653279"/' -i
    /etc/default/cloudera-scm-agent *
    *END (0) *
    *BEGIN grep server_host=ip-10-124-243-21.ec2.internal
    /etc/cloudera-scm-agent/config.ini *
    server_host=ip-10-124-243-21.**e**c2.internal
    *END (0) *
    scm agent is already configured
    Starting Cloudera Manager Agent...
    *BEGIN /sbin/service cloudera-scm-agent status | grep running *
    *END (1) *
    *BEGIN /sbin/service cloudera-scm-agent start *
    Starting cloudera-scm-agent: [60G[ [0;31mFAILED [0;39m]
    *END (1) *
    agent logs:
    *BEGIN tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.out
    sed 's/^/>>/' *
    /usr/lib64/cmf/agent/src/**cmf**/agent.py:31: DeprecationWarning:
    the sha module is deprecated; use the hashlib module instead
    import sha
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO SCM Agent
    Version: 4.6.0
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Using
    directory: /var/run/cloudera-scm-agent
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Using
    supervisor binary path: /usr/lib64/cmf/agent/src/cmf/.****
    ./../build/env/bin/**supervisord**
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Adding env
    vars that start with CMF_AGENT_
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: HIVE_DEFAULT_XML: /etc/hive/conf.dist/hive-
    **defau**lt.xml
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: MGMT_HOME: /usr/share/cmf
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HTTPFS_HOME: /usr/lib/hadoop-httpfs
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_FLUME_HOME: /usr/lib/flume-ng
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HBASE_HOME: /usr/lib/hbase
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HUE_HOME: /usr/share/hue
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: TOMCAT_HOME: /usr/lib/bigtop-tomcat
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_PIG_HOME: /usr/lib/pig
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HIVE_HOME: /usr/lib/hive
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_YARN_HOME: /usr/lib/hadoop-yarn
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HADOOP_BIN: /usr/bin/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_OOZIE_HOME: /usr/lib/oozie
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_ZOOKEEPER_HOME: /usr/lib/zookeeper
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HUE_PLUGINS_HOME: /usr/lib/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: JSVC_HOME: /usr/libexec/bigtop-utils
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HDFS_HOME: /usr/lib/hadoop-hdfs
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_MR1_HOME: /usr/lib/hadoop-0.20-**
    mapreduce**
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_MR2_HOME: /usr/lib/hadoop-mapreduce
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HADOOP_HOME: /usr/lib/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_IMPALA_HOME: /usr/lib/impala
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Logging to
    /var/log/cloudera-scm-agent/**cl**oudera-scm-agent.log
    Traceback (most recent call last):
    File "/usr/lib64/cmf/agent/src/cmf/****agent.py", line 2391, in
    <module>
    main()
    File "/usr/lib64/cmf/agent/src/cmf/****agent.py", line 2375, in
    main
    agent.start()
    File "/usr/lib64/cmf/agent/src/cmf/****agent.py", line 329, in
    start
    self.add_db_driver_jars()
    File "/usr/lib64/cmf/agent/src/cmf/****agent.py", line 1400, in
    add_db_driver_ja>>/usr/lib64/**c**mf/agent/src/cmf/agent.py:31:
    DeprecationWarning: the sha module is deprecated; use the hashlib module
    instead
    import sha
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO SCM Agent
    Version: 4.6.0
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Using
    directory: /var/run/cloudera-scm-agent
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Using
    supervisor binary path: /usr/lib64/cmf/agent/src/cmf/.****
    ./../build/env/bin/**supervisord**
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Adding env
    vars that start with CMF_AGENT_
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: HIVE_DEFAULT_XML: /etc/hive/conf.dist/hive-
    **defau**lt.xml
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: MGMT_HOME: /usr/share/cmf
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HTTPFS_HOME: /usr/lib/hadoop-httpfs
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_FLUME_HOME: /usr/lib/flume-ng
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HBASE_HOME: /usr/lib/hbase
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HUE_HOME: /usr/share/hue
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: TOMCAT_HOME: /usr/lib/bigtop-tomcat
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_PIG_HOME: /usr/lib/pig
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HIVE_HOME: /usr/lib/hive
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_YARN_HOME: /usr/lib/hadoop-yarn
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HADOOP_BIN: /usr/bin/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_OOZIE_HOME: /usr/lib/oozie
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_ZOOKEEPER_HOME: /usr/lib/zookeeper
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HUE_PLUGINS_HOME: /usr/lib/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: JSVC_HOME: /usr/libexec/bigtop-utils
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HDFS_HOME: /usr/lib/hadoop-hdfs
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_MR1_HOME: /usr/lib/hadoop-0.20-**
    mapreduce**
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_MR2_HOME: /usr/lib/hadoop-mapreduce
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_HADOOP_HOME: /usr/lib/hadoop
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO explicitly
    passing env var to supervisee: CDH_IMPALA_HOME: /usr/lib/impala
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO Logging to
    /var/log/cloudera-scm-agent/**cl**oudera-scm-agent.log
    Traceback (most recent call last):
    File "/usr/lib64/cmf/agent/src/cmf/****agent.py", line 2391, in
    <module>
    main()
    File "/usr/lib64/cmf/agent/src/cmf/****agent.py", line 2375, in
    main
    agent.start()
    File "/usr/lib64/cmf/agent/src/cmf/****agent.py", line 329, in
    start
    self.add_db_driver_jars()
    File "/usr/lib64/cmf/agent/src/cmf/****agent.py", line 1400, in
    add_db_driver_jars
    for jar in (mysql_jar, oracle_jar, postgresql_jar):
    UnboundLocalError: local variable 'postgresql_jar' referenced before
    assignment
    rs
    for jar in (mysql_jar, oracle_jar, postgresql_jar):
    UnboundLocalError: local variable 'postgresql_jar' referenced before
    assignment
    *END (0) *
    *BEGIN tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.log
    sed 's/^/>>/' *
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Deleting
    process 1014-mapreduce-TASKTRACKER
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Triggering
    supervisord update.
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Retiring
    process 1014-mapreduce-TASKTRACKER
    [07/Jun/2013 22:59:32 +0000] 695 MonitorDaemon-Reporter
    throttling_logger ERROR Error sending messages to firehose:
    mgmt1-HOSTMONITOR-**d889e476015d**035afbfed2002d76de**2d
    Traceback (most recent call last):
    File "/usr/lib64/cmf/agent/src/cmf/****monitor/firehose.py", line
    67, in send
    self._requestor.request('**sendA**gentMessages',
    dict(messages=messages))
    File "/usr/lib64/cmf/agent/build/**en**v/lib/python2.6/site-**
    packages/**avro-1.6.3-py2.6.egg/**avro/ipc.**py", line 145, in
    request
    return self.issue_request(call_**reques**t, message_name,
    request_datum)
    File "/usr/lib64/cmf/agent/build/**en**v/lib/python2.6/site-**
    packages/**avro-1.6.3-py2.6.egg/**avro/ipc.**py", line 256, in
    issue_request
    call_response = self.transceiver.transceive(**ca**ll_request)
    File "/usr/lib64/cmf/agent/build/**en**v/lib/python2.6/site-**
    packages/**avro-1.6.3-py2.6.egg/**avro/ipc.**py", line 485, in
    transceive
    result = self.read_framed_message()
    File "/usr/lib64/cmf/agent/build/**en**v/lib/python2.6/site-**
    packages/**avro-1.6.3-py2.6.egg/**avro/ipc.**py", line 489, in
    read_framed_message
    response = self.conn.getresponse()
    File "/usr/lib64/python2.6/httplib.****py", line 990, in
    getresponse
    response.begin()
    File "/usr/lib64/python2.6/httplib.****py", line 391, in begin
    version, status, reason = self._read_status()
    File "/usr/lib64/python2.6/httplib.****py", line 355, in
    _read_status
    raise BadStatusLine(line)
    BadStatusLine
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Stopping
    agent...
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO No cgrouped
    processes running; unmounting any cgroup roots
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO No processes
    are being managed; Supervisor will shutdown.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Shutting down
    supervisord, pid 715
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Successfully
    shutdown supervisord, pid 715
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE HTTP Server cherrypy._cpwsgi_server.**
    CPWSGI**Server(('dn6.lab10.**adconion.**com<http://dn6.lab10.adconion.com>',
    9000)) shut down
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Stopped thread '_TimeoutMonitor'.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE HTTP Server cherrypy._cpwsgi_server.**
    CPWSGI**Server(('dn6.lab10.**adconion.**com<http://dn6.lab10.adconion.com>',
    9000)) already shut down
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE No thread running for None.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus EXITING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus EXITED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Agent
    exiting; caught signal 15
    [07/Jun/2013 23:02:30 +0000] 12620 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:02:30 +0000] 12620 Dummy-1 agent INFO Stopping
    agent...
    [07/Jun/2013 23:20:25 +0000] 14136 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:20:25 +0000] 14136 Dummy-1 agent INFO Stopping
    agent...
    [07/Jun/2013 23:51:20 +0000] 15580 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:51:20 +0000] 15580 Dummy-1 agent INFO Stopping
    agent...
    [07/Jun/2013 23:53:21 +0000] 16382 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:53:21 +0000] 16382 Dummy-1 agent INFO Stopping
    agent...
    [08/Jun/2013 00:05:13 +0000] 17261 MainThread agent INFO No command
    line vars
    [08/Jun/2013 00:05:13 +0000] 17261 Dummy-1 agent INFO Stopping
    agent...
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO No command
    line vars
    [08/Jun/2013 00:43:29 +0000] 20804 Dummy-1 agent INFO Stopping
    agent...
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Deleting
    process 1014-mapreduce-TASKTRACKER
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Triggering
    supervisord update.
    [07/Jun/2013 22:58:49 +0000] 695 MainThread agent INFO Retiring
    process 1014-mapreduce-TASKTRACKER
    [07/Jun/2013 22:59:32 +0000] 695 MonitorDaemon-Reporter
    throttling_logger ERROR Error sending messages to firehose:
    mgmt1-HOSTMONITOR-**d889e476015d**035afbfed2002d76de**2d
    Traceback (most recent call last):
    File "/usr/lib64/cmf/agent/src/cmf/****monitor/firehose.py", line
    67, in send
    self._requestor.request('**sendA**gentMessages',
    dict(messages=messages))
    File "/usr/lib64/cmf/agent/build/**en**v/lib/python2.6/site-**
    packages/**avro-1.6.3-py2.6.egg/**avro/ipc.**py", line 145, in
    request
    return self.issue_request(call_**reques**t, message_name,
    request_datum)
    File "/usr/lib64/cmf/agent/build/**en**v/lib/python2.6/site-**
    packages/**avro-1.6.3-py2.6.egg/**avro/ipc.**py", line 256, in
    issue_request
    call_response = self.transceiver.transceive(**ca**ll_request)
    File "/usr/lib64/cmf/agent/build/**en**v/lib/python2.6/site-**
    packages/**avro-1.6.3-py2.6.egg/**avro/ipc.**py", line 485, in
    transceive
    result = self.read_framed_message()
    File "/usr/lib64/cmf/agent/build/**en**v/lib/python2.6/site-**
    packages/**avro-1.6.3-py2.6.egg/**avro/ipc.**py", line 489, in
    read_framed_message
    response = self.conn.getresponse()
    File "/usr/lib64/python2.6/httplib.****py", line 990, in
    getresponse
    response.begin()
    File "/usr/lib64/python2.6/httplib.****py", line 391, in begin
    version, status, reason = self._read_status()
    File "/usr/lib64/python2.6/httplib.****py", line 355, in
    _read_status
    raise BadStatusLine(line)
    BadStatusLine
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Stopping
    agent...
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO No cgrouped
    processes running; unmounting any cgroup roots
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO No processes
    are being managed; Supervisor will shutdown.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Shutting down
    supervisord, pid 715
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Successfully
    shutdown supervisord, pid 715
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE HTTP Server cherrypy._cpwsgi_server.**
    CPWSGI**Server(('dn6.lab10.**adconion.**com<http://dn6.lab10.adconion.com>',
    9000)) shut down
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Stopped thread '_TimeoutMonitor'.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE HTTP Server cherrypy._cpwsgi_server.**
    CPWSGI**Server(('dn6.lab10.**adconion.**com<http://dn6.lab10.adconion.com>',
    9000)) already shut down
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE No thread running for None.
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus STOPPED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus EXITING
    [07/Jun/2013 23:02:27 +0000] 695 MainThread _cplogging INFO
    [07/Jun/2013:23:02:27] ENGINE Bus EXITED
    [07/Jun/2013 23:02:27 +0000] 695 MainThread agent INFO Agent
    exiting; caught signal 15
    [07/Jun/2013 23:02:30 +0000] 12620 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:02:30 +0000] 12620 Dummy-1 agent INFO Stopping
    agent...
    [07/Jun/2013 23:20:25 +0000] 14136 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:20:25 +0000] 14136 Dummy-1 agent INFO Stopping
    agent...
    [07/Jun/2013 23:51:20 +0000] 15580 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:51:20 +0000] 15580 Dummy-1 agent INFO Stopping
    agent...
    [07/Jun/2013 23:53:21 +0000] 16382 MainThread agent INFO No command
    line vars
    [07/Jun/2013 23:53:21 +0000] 16382 Dummy-1 agent INFO Stopping
    agent...
    [08/Jun/2013 00:05:13 +0000] 17261 MainThread agent INFO No command
    line vars
    [08/Jun/2013 00:05:13 +0000] 17261 Dummy-1 agent INFO Stopping
    agent...
    [08/Jun/2013 00:43:29 +0000] 20804 MainThread agent INFO No command
    line vars
    [08/Jun/2013 00:43:29 +0000] 20804 Dummy-1 agent INFO Stopping
    agent...
    *END (0) *
    end of agent logs.
    scm agent could not be started, giving up
    waiting for rollback request

    I hope you can help.

    Thanks,
    Ben
    On Friday, June 7, 2013 3:55:09 PM UTC-7, Benjamin Kim wrote:

    I just upgraded the CM server in EC2 to 4.6 from 4.5.2. I am trying
    to upgrade the node agents to the 4.6 using the wizard. I have 11 EC2 nodes
    in the cluster. When it gets to the Upgrade Hosts page, it cannot connect
    to any of the nodes in EC2. The names of all the nodes have been replaced
    with names like "i-4f792e33". I can see the original hosts with
    their correct names on the Hosts page. What can I do to fix this?

    Thanks,
    Ben

    --
    Thanks,
    Darren

    --
    Thanks,
    Darren

    --
    Thanks,
    Darren

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedJun 7, '13 at 10:55p
activeJun 10, '13 at 8:57p
posts7
users2
websitecloudera.com
irc#hadoop

2 users in discussion

Benjamin Kim: 4 posts Darren Lo: 3 posts

Content

People

Support

Translate

site design / logo © 2022 Grokbase