FAQ
Hi Guys-

I am setting up a HBase cluster on Amazon EC2 that is reachable externally
from the internet, so I can run jars directly from eclipse on my
development machine.

I have SCM server installed on Amazon EC2 cloud machine, with an elastic IP
of 23.23.192.197

Hostname:

[root@54 cloudera-scm-server]# vi /etc/hosts
[root@54 cloudera-scm-server]# hostname
ec2-23-23-192-197.compute-1.amazonaws.com

Machine can ping its external IP and resolves correctly:

[root@54 cloudera-scm-server]# ping
ec2-23-23-192-197.compute-1.amazonaws.com
PING ec2-23-23-192-197.compute-1.amazonaws.com (23.23.192.197) 56(84) bytes
of data.
64 bytes from ec2-23-23-192-197.compute-1.amazonaws.com (23.23.192.197):
icmp_seq=1 ttl=51 time=1.03 ms
64 bytes from ec2-23-23-192-197.compute-1.amazonaws.com (23.23.192.197):
icmp_seq=2 ttl=51 time=0.976 ms
64 bytes from ec2-23-23-192-197.compute-1.amazonaws.com (23.23.192.197):
icmp_seq=3 ttl=51 time=1.09 ms
64 bytes from ec2-23-23-192-197.compute-1.amazonaws.com (23.23.192.197):
icmp_seq=4 ttl=51 time=0.936 ms
^C
--- ec2-23-23-192-197.compute-1.amazonaws.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3154ms
rtt min/avg/max/mdev = 0.936/1.010/1.097/0.060 ms

I didn't use EC2 Cloud and rather use the classical hosts wizard. I edited
the cloudera agent config.ini to use both the external IP and DNS for agent
to heartbeat to cloudera manager, however the agent is unable to reach into
the server.

[root@54 cloudera-scm-server]# vi /etc/cloudera-scm-agent/
[root@54 cloudera-scm-server]# vi /etc/cloudera-scm-agent/config.ini
[root@54 cloudera-scm-server]# service cloudera-scm-agent start
Starting cloudera-scm-agent: [FAILED]
[root@54 cloudera-scm-server]#
The log is indicating: error: No socket could be created on
('ec2-23-23-192-197.compute-1.amazonaws.com', 9000) -- [Errno 99] Cannot
assign requested address

I have opened all ports from outside to connect on external IP but the
issue persists,

How do i get past this warning? The goal is to install via the classical
SCM wizard for the cluster to be open to the internet. I was able to
install the same cluster with local IP on a separate node, which is not
reachable from internet, however when i run the code from eclipse from dev
machine coming over internet, it fails to resolve HMaster and ZOOK, since
they are on internal IP.

Any help from users or Cloudera/BC Wong would be highly appreciated.
Thanks
Sal

Search Discussions

  • Darren Lo at Jun 4, 2013 at 5:05 pm
    Hi Salman,

    From the error, it sounds like you have a port conflict. Is something else
    listening on port 9000?

    Thanks,
    Darren

    On Tue, Jun 4, 2013 at 7:04 AM, Salman Ahmed wrote:

    Hi Guys-

    I am setting up a HBase cluster on Amazon EC2 that is reachable externally
    from the internet, so I can run jars directly from eclipse on my
    development machine.

    I have SCM server installed on Amazon EC2 cloud machine, with an elastic
    IP of 23.23.192.197

    Hostname:

    [root@54 cloudera-scm-server]# vi /etc/hosts
    [root@54 cloudera-scm-server]# hostname
    ec2-23-23-192-197.compute-1.amazonaws.com

    Machine can ping its external IP and resolves correctly:

    [root@54 cloudera-scm-server]# ping
    ec2-23-23-192-197.compute-1.amazonaws.com
    PING ec2-23-23-192-197.compute-1.amazonaws.com (23.23.192.197) 56(84)
    bytes of data.
    64 bytes from ec2-23-23-192-197.compute-1.amazonaws.com (23.23.192.197):
    icmp_seq=1 ttl=51 time=1.03 ms
    64 bytes from ec2-23-23-192-197.compute-1.amazonaws.com (23.23.192.197):
    icmp_seq=2 ttl=51 time=0.976 ms
    64 bytes from ec2-23-23-192-197.compute-1.amazonaws.com (23.23.192.197):
    icmp_seq=3 ttl=51 time=1.09 ms
    64 bytes from ec2-23-23-192-197.compute-1.amazonaws.com (23.23.192.197):
    icmp_seq=4 ttl=51 time=0.936 ms
    ^C
    --- ec2-23-23-192-197.compute-1.amazonaws.com ping statistics ---
    4 packets transmitted, 4 received, 0% packet loss, time 3154ms
    rtt min/avg/max/mdev = 0.936/1.010/1.097/0.060 ms

    I didn't use EC2 Cloud and rather use the classical hosts wizard. I edited
    the cloudera agent config.ini to use both the external IP and DNS for agent
    to heartbeat to cloudera manager, however the agent is unable to reach into
    the server.

    [root@54 cloudera-scm-server]# vi /etc/cloudera-scm-agent/
    [root@54 cloudera-scm-server]# vi /etc/cloudera-scm-agent/config.ini
    [root@54 cloudera-scm-server]# service cloudera-scm-agent start
    Starting cloudera-scm-agent: [FAILED]
    [root@54 cloudera-scm-server]#
    The log is indicating: error: No socket could be created on ('
    ec2-23-23-192-197.compute-1.amazonaws.com', 9000) -- [Errno 99] Cannot
    assign requested address

    I have opened all ports from outside to connect on external IP but the
    issue persists,

    How do i get past this warning? The goal is to install via the classical
    SCM wizard for the cluster to be open to the internet. I was able to
    install the same cluster with local IP on a separate node, which is not
    reachable from internet, however when i run the code from eclipse from dev
    machine coming over internet, it fails to resolve HMaster and ZOOK, since
    they are on internal IP.

    Any help from users or Cloudera/BC Wong would be highly appreciated.
    Thanks
    Sal


    --
    Thanks,
    Darren
  • Salman Ahmed at Jun 4, 2013 at 6:02 pm
    Just checked netstat -an | grep -i 9000, nothing is running.
    Thanks
    On Tuesday, June 4, 2013 10:05:31 AM UTC-7, Darren Lo wrote:

    Hi Salman,

    From the error, it sounds like you have a port conflict. Is something else
    listening on port 9000?

    Thanks,
    Darren


    On Tue, Jun 4, 2013 at 7:04 AM, Salman Ahmed <ahmed....@gmail.com<javascript:>
    wrote:
    Hi Guys-

    I am setting up a HBase cluster on Amazon EC2 that is reachable
    externally from the internet, so I can run jars directly from eclipse on my
    development machine.

    I have SCM server installed on Amazon EC2 cloud machine, with an elastic
    IP of 23.23.192.197

    Hostname:

    [root@54 cloudera-scm-server]# vi /etc/hosts
    [root@54 cloudera-scm-server]# hostname
    ec2-23-23-192-197.compute-1.amazonaws.com

    Machine can ping its external IP and resolves correctly:

    [root@54 cloudera-scm-server]# ping
    ec2-23-23-192-197.compute-1.amazonaws.com
    PING ec2-23-23-192-197.compute-1.amazonaws.com (23.23.192.197) 56(84)
    bytes of data.
    64 bytes from ec2-23-23-192-197.compute-1.amazonaws.com (23.23.192.197):
    icmp_seq=1 ttl=51 time=1.03 ms
    64 bytes from ec2-23-23-192-197.compute-1.amazonaws.com (23.23.192.197):
    icmp_seq=2 ttl=51 time=0.976 ms
    64 bytes from ec2-23-23-192-197.compute-1.amazonaws.com (23.23.192.197):
    icmp_seq=3 ttl=51 time=1.09 ms
    64 bytes from ec2-23-23-192-197.compute-1.amazonaws.com (23.23.192.197):
    icmp_seq=4 ttl=51 time=0.936 ms
    ^C
    --- ec2-23-23-192-197.compute-1.amazonaws.com ping statistics ---
    4 packets transmitted, 4 received, 0% packet loss, time 3154ms
    rtt min/avg/max/mdev = 0.936/1.010/1.097/0.060 ms

    I didn't use EC2 Cloud and rather use the classical hosts wizard. I
    edited the cloudera agent config.ini to use both the external IP and DNS
    for agent to heartbeat to cloudera manager, however the agent is unable to
    reach into the server.

    [root@54 cloudera-scm-server]# vi /etc/cloudera-scm-agent/
    [root@54 cloudera-scm-server]# vi /etc/cloudera-scm-agent/config.ini
    [root@54 cloudera-scm-server]# service cloudera-scm-agent start
    Starting cloudera-scm-agent: [FAILED]
    [root@54 cloudera-scm-server]#
    The log is indicating: error: No socket could be created on ('
    ec2-23-23-192-197.compute-1.amazonaws.com', 9000) -- [Errno 99] Cannot
    assign requested address

    I have opened all ports from outside to connect on external IP but the
    issue persists,

    How do i get past this warning? The goal is to install via the classical
    SCM wizard for the cluster to be open to the internet. I was able to
    install the same cluster with local IP on a separate node, which is not
    reachable from internet, however when i run the code from eclipse from dev
    machine coming over internet, it fails to resolve HMaster and ZOOK, since
    they are on internal IP.

    Any help from users or Cloudera/BC Wong would be highly appreciated.
    Thanks
    Sal


    --
    Thanks,
    Darren
  • Salman Ahmed at Jun 4, 2013 at 7:22 pm
    Some more log snippets from cloudera agent. I killed all rogue agents, and
    put chkconfig for cloudera agent


    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO Re-using
    pre-existing directory: /var/run/cloudera-scm-agent/process
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO Re-using
    pre-existing directory: /var/run/cloudera-scm-agent/supervisor
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO Re-using
    pre-existing directory: /var/run/cloudera-scm-agent/supervisor/include
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO Failed
    to connect to previous supervisor.
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread agent INFO Trying
    to connect to newly launched supervisor (Attempt 1)
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread agent INFO
    Successfully connected to supervisor
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread _cplogging INFO
    [04/Jun/2013:15:17:27] ENGINE Bus STARTING
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread _cplogging INFO
    [04/Jun/2013:15:17:27] ENGINE Started monitor thread '_TimeoutMonitor'.
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging ERROR
    [04/Jun/2013:15:17:27] ENGINE Error in HTTP server: shutting down
    Traceback (most recent call last):
       File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/process/servers.py",
    line 187, in _start_http_thread
         self.httpserver.start()
       File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/wsgiserver/wsgiserver2.py",
    line 1825, in start
         raise socket.error(msg)
    error: No socket could be created on ('
    ec2-23-23-192-197.compute-1.amazonaws.com', 9000) -- [Errno 99] Cannot
    assign requested address


    [root@ec2-23-23-192-197 cloudera-scm-agent]# cat cloudera-scm-agent.log
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO No
    command line vars
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Agent
    starting as pid 1511 user root(0) group root(0).
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Re-using
    pre-existing directory: /var/run/cloudera-scm-agent
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread downloader INFO
    Downloader path: /opt/cloudera/parcel-cache
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread parcel_cache INFO Using
    /opt/cloudera/parcel-cache for parcel cache
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Re-using
    pre-existing directory:
    /var/run/cloudera-scm-agent/cg
    roups
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread cgroups INFO Reusing
    /var/run/cloudera-scm-agent/cgroups/memory
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread cgroups INFO Reusing
    /var/run/cloudera-scm-agent/cgroups/cpu
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread cgroups INFO Reusing
    /var/run/cloudera-scm-agent/cgroups/blkio
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Found
    cgroups capabilities: {'has_memory': True,
    'default_memor
    y_limit_in_bytes': 9223372036854775807, 'writable_cgroup_dot_procs': True,
    'has_cpu': True, 'default_blkio_weight': 1000,
    'default _cpu_shares': 1024,
    'has_blkio': True}
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Setting
    up supervisord event monitor.
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread filesystem_map INFO
    Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs']
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread filesystem_map INFO Using
    timeout of 2.000000
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread filesystem_map INFO Using
    join timeout of 0.100000
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread filesystem_map INFO Using
    tolerance of 60.000000
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Using
    metrics_url_timeout_seconds of 30.000000
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Using
    task_metrics_timeout_seconds of 5.000000
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Using
    max_collection_wait_seconds of 10.000000
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread metrics INFO
    Importing tasktracker metric schema from file
    /usr/lib64/cmf/ag
    ent/src/cmf/monitor/tasktracker/schema.json
    [04/Jun/2013 15:17:24 +0000] 1511 MainThread __init__ INFO
    Importing metric schema from file
    /usr/lib64/cmf/agent/src/cmf/
    monitor/schema.json
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Supervised processes will inherit the following environment:
    {' CMF_CONF_DIR':
    '/etc/cloudera-scm-agent', 'CMF_AGENT_HIVE_DEFAULT_XML':
    '/etc/hive/conf.dist/hive-default.xml',
    'CMF_AGENT_MGMT_HO ME':
    '/usr/share/cmf', 'CMF_AGENT_CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs',
    'PATH': '/sbin:/usr/sbin:/bin:/usr/bin',
    'CMF_AGENT_ CDH_FLUME_HOME':
    '/usr/lib/flume-ng', 'CMF_AGENT_CDH_HBASE_HOME': '/usr/lib/hbase', 'LANG':
    'en_US.UTF-8', 'TERM': 'xterm',
    'CLOUD
    ERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar',
    'CMF_AGENT_CDH_HUE_HOME': '/usr/share/hue', 'SHLVL':
    '1',
    'CMF_AGENT_TOMCAT_HOME': '/usr/lib/bigtop-tomcat',
    'CLOUDERA_POSTGRESQL_JDBC_JAR':
    '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.ja
    r', 'CMF_AGENT_CDH_PIG_HOME': '/usr/lib/pig', 'CMF_AGENT_CDH_HIVE_HOME':
    '/usr/lib/hive', 'CLOUDERA_MYSQL_CONNECTOR_JAR':
    '/usr/sh
    are/java/mysql-connector-java.jar', 'CMF_AGENT_CDH_YARN_HOME':
    '/usr/lib/hadoop-yarn', 'CMF_AGENT_CDH_HADOOP_BIN':
    '/usr/bin/hadoo p',
    'CMF_AGENT_CDH_OOZIE_HOME': '/usr/lib/oozie',
    'CMF_AGENT_CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper',
    'CMF_AGENT_CDH_HUE_PLUGINS
    _HOME': '/usr/lib/hadoop', 'CMF_AGENT_JSVC_HOME':
    '/usr/libexec/bigtop-utils', 'CMF_AGENT_CDH_HDFS_HOME':
    '/usr/lib/hadoop-hdfs',
    'CMF_AGENT_CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce',
    'CMF_AGENT_CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'PWD': '/',
    'CM F_AGENT_CDH_HADOOP_HOME':
    '/usr/lib/hadoop', 'CMF_AGENT_CDH_IMPALA_HOME': '/usr/lib/impala'}
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Supervised processes will add the following their environment
    e nvironment:
    {'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'CMF_PACKAGE_DIR':
    '/usr/lib64/cmf/service', 'CDH_HADOOP_BIN':
    '/usr/bin /hadoop',
    'MGMT_HOME': '/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala',
    'CDH_YARN_HOME': '/usr/lib/hadoop-yarn',
    'CDH_HDFS_ HOME':
    '/usr/lib/hadoop-hdfs', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop',
    'CM_STATUS_CODES': u'STATUS_NONE
    HDFS_DFS_DIR_NOT_EMPTY' ,
    'CLOUDERA_ORACLE_CONNECTOR_JAR':
    '/usr/share/java/oracle-connector-java.jar', 'CDH_HIVE_HOME':
    '/usr/lib/hive', 'JSVC_HOME':
    '/u sr/libexec/bigtop-utils',
    'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml',
    'CLOUDERA_POSTGRESQL_JDBC_JAR':
    '/usr/share/
    cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_HBASE_HOME': '/usr/lib/hbase',
    'CDH_OOZIE_HOME': '/usr/lib/oozie',
    'CDH_ZOOKEEPER_HOME ':
    '/usr/lib/zookeeper', 'CDH_HUE_HOME': '/usr/share/hue',
    'CLOUDERA_MYSQL_CONNECTOR_JAR':
    '/usr/share/java/mysql-connector-java.j
    ar', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_PIG_HOME':
    '/usr/lib/pig', 'CDH_HTTPFS_HOME':
    '/usr/lib/hadoop-httpfs' ,
    'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'TOMCAT_HOME':
    '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'}
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO Re-using
    pre-existing directory:
    /var/run/cloudera-scm-agent/pr
    ocess
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO Re-using
    pre-existing directory:
    /var/run/cloudera-scm-agent/su
    pervisor
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO Re-using
    pre-existing directory:
    /var/run/cloudera-scm-agent/su
    pervisor/include
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO Failed
    to connect to previous supervisor.
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread agent INFO Trying
    to connect to newly launched supervisor (Attempt 1)
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread agent INFO
    Successfully connected to supervisor
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread _cplogging INFO
    [04/Jun/2013:15:17:27] ENGINE Bus STARTING
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread _cplogging INFO
    [04/Jun/2013:15:17:27] ENGINE Started monitor thread
    '_TimeoutM onitor'.
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging ERROR
    [04/Jun/2013:15:17:27] ENGINE Error in HTTP server:
    sh utting down
    Traceback (most recent call last):
       File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/process/servers.py",
    line 187 , in
    _start_http_thread
         self.httpserver.start()
       File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/wsgiserver/wsgiserver2.py",
    l ine 1825, in start
         raise socket.error(msg)
    error: No socket could be created on ('
    ec2-23-23-192-197.compute-1.amazonaws.com', 9000) -- [Errno 99] Cannot
    assign requested add ress

    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging INFO
    [04/Jun/2013:15:17:27] ENGINE Bus STOPPING
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging INFO
    [04/Jun/2013:15:17:27] ENGINE HTTP Server
    cherrypy._cp
    wsgi_server.CPWSGIServer(('ec2-23-23-192-197.compute-1.amazonaws.com',
    9000)) already shut down
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging INFO
    [04/Jun/2013:15:17:27] ENGINE Stopped thread
    '_Timeout Monitor'.
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging INFO
    [04/Jun/2013:15:17:27] ENGINE Bus STOPPED
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging INFO
    [04/Jun/2013:15:17:27] ENGINE Bus EXITING
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging INFO
    [04/Jun/2013:15:17:27] ENGINE Bus EXITED
    [root@ec2-23-23-192-197 cloudera-scm-agent]# cat cloudera-scm-agent.out
    /usr/lib64/cmf/agent/src/cmf/agent.py:31: DeprecationWarning: the sha
    module is deprecated; use the hashlib module instead
       import sha
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO SCM
    Agent Version: 4.5.3
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Missing
    database jar: /usr/share/java/mysql-connector-java.jar (normal, if you're
    not using this database type)
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Missing
    database jar: /usr/share/java/oracle-connector-java.jar (normal, if you're
    not using this database type)
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Found
    database jar: /usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Using
    directory: /var/run/cloudera-scm-agent
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Using
    supervisor binary path:
    /usr/lib64/cmf/agent/src/cmf/../../build/env/bin/supervisord
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Adding
    env vars that start with CMF_AGENT_
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: HIVE_DEFAULT_XML:
    /etc/hive/conf.dist/hive-default.xml
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: MGMT_HOME: /usr/share/cmf
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HTTPFS_HOME:
    /usr/lib/hadoop-httpfs
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_FLUME_HOME: /usr/lib/flume-ng
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HBASE_HOME: /usr/lib/hbase
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HUE_HOME: /usr/share/hue
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: TOMCAT_HOME:
    /usr/lib/bigtop-tomcat
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_PIG_HOME: /usr/lib/pig
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HIVE_HOME: /usr/lib/hive
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_YARN_HOME:
    /usr/lib/hadoop-yarn
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HADOOP_BIN: /usr/bin/hadoop
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_OOZIE_HOME: /usr/lib/oozie
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_ZOOKEEPER_HOME:
    /usr/lib/zookeeper
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HUE_PLUGINS_HOME:
    /usr/lib/hadoop
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: JSVC_HOME:
    /usr/libexec/bigtop-utils
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HDFS_HOME:
    /usr/lib/hadoop-hdfs
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_MR1_HOME:
    /usr/lib/hadoop-0.20-mapreduce
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_MR2_HOME:
    /usr/lib/hadoop-mapreduce
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HADOOP_HOME: /usr/lib/hadoop
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_IMPALA_HOME: /usr/lib/impala
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Logging
    to /var/log/cloudera-scm-agent/cloudera-scm-agent.log
    Error: could not find config file
    /var/run/cloudera-scm-agent/supervisor/supervisord.conf
    For help, use /usr/lib64/cmf/agent/src/cmf/agent.py -h




    On Tue, Jun 4, 2013 at 11:02 AM, Salman Ahmed wrote:

    Just checked netstat -an | grep -i 9000, nothing is running.
    Thanks

    On Tuesday, June 4, 2013 10:05:31 AM UTC-7, Darren Lo wrote:

    Hi Salman,

    From the error, it sounds like you have a port conflict. Is something
    else listening on port 9000?

    Thanks,
    Darren

    On Tue, Jun 4, 2013 at 7:04 AM, Salman Ahmed wrote:

    Hi Guys-

    I am setting up a HBase cluster on Amazon EC2 that is reachable
    externally from the internet, so I can run jars directly from eclipse on my
    development machine.

    I have SCM server installed on Amazon EC2 cloud machine, with an elastic
    IP of 23.23.192.197

    Hostname:

    [root@54 cloudera-scm-server]# vi /etc/hosts
    [root@54 cloudera-scm-server]# hostname ec2-23-23-192-197.compute-1.**
    amazonaws.com <http://ec2-23-23-192-197.compute-1.amazonaws.com>

    Machine can ping its external IP and resolves correctly:

    [root@54 cloudera-scm-server]# ping ec2-23-23-192-197.compute-1.**
    amazonaws.com <http://ec2-23-23-192-197.compute-1.amazonaws.com>
    PING ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>(23.23.192.197) 56(84) bytes of data.
    64 bytes from ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>(23.23.192.197): icmp_seq=1 ttl=51 time=1.03 ms
    64 bytes from ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>(23.23.192.197): icmp_seq=2 ttl=51 time=0.976 ms
    64 bytes from ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>(23.23.192.197): icmp_seq=3 ttl=51 time=1.09 ms
    64 bytes from ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>(23.23.192.197): icmp_seq=4 ttl=51 time=0.936 ms
    ^C
    --- ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>ping statistics ---
    4 packets transmitted, 4 received, 0% packet loss, time 3154ms
    rtt min/avg/max/mdev = 0.936/1.010/1.097/0.060 ms

    I didn't use EC2 Cloud and rather use the classical hosts wizard. I
    edited the cloudera agent config.ini to use both the external IP and DNS
    for agent to heartbeat to cloudera manager, however the agent is unable to
    reach into the server.

    [root@54 cloudera-scm-server]# vi /etc/cloudera-scm-agent/
    [root@54 cloudera-scm-server]# vi /etc/cloudera-scm-agent/**config.ini
    [root@54 cloudera-scm-server]# service cloudera-scm-agent start
    Starting cloudera-scm-agent: [FAILED]
    [root@54 cloudera-scm-server]#
    The log is indicating: error: No socket could be created on ('
    ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>',
    9000) -- [Errno 99] Cannot assign requested address

    I have opened all ports from outside to connect on external IP but the
    issue persists,

    How do i get past this warning? The goal is to install via the classical
    SCM wizard for the cluster to be open to the internet. I was able to
    install the same cluster with local IP on a separate node, which is not
    reachable from internet, however when i run the code from eclipse from dev
    machine coming over internet, it fails to resolve HMaster and ZOOK, since
    they are on internal IP.

    Any help from users or Cloudera/BC Wong would be highly appreciated.
    Thanks
    Sal


    --
    Thanks,
    Darren
  • Salman Ahmed at Jun 4, 2013 at 8:11 pm
    So when i put the following mapping in /etc/hosts, due to last line
    cloudera agent doesnt start->
    [root@ec2-23-23-192-197 cloudera-scm-agent]# cat /etc/hosts
    127.0.0.1 localhost.localdomain localhost
    ::1 localhost6.localdomain6 localhost6
    23.23.192.197 ec2-23-23-192-197.compute-1.amazonaws.com


    Nslookup is revealing some interesting quirks. We may need to map DNS and
    hosts resolution that is IP 23.23.192.197 always resolves to
    ec2-23-23-192-197.compute-1.amazonaws.com AND
    ec2-23-23-192-197.compute-1.amazonaws.com always resolve to 23.23.192.197
    [root@ec2-23-23-192-197 cloudera-scm-agent]# nslookup 23.23.192.197
    Server: 172.16.0.23
    Address: 172.16.0.23#53

    Non-authoritative answer:
    197.192.23.23.in-addr.arpa
       name = ec2-23-23-192-197.compute-1.amazonaws.com.

    Authoritative answers can be found from:

    [root@ec2-23-23-192-197 cloudera-scm-agent]# nslookup
    ec2-23-23-192-197.compute-1.amazonaws.com
    Server: 172.16.0.23
    Address: 172.16.0.23#53

    Non-authoritative answer:
    Name: ec2-23-23-192-197.compute-1.amazonaws.com
    Address: 10.145.232.83 <---- This should resolve to 23.23.192.197???

    [root@ec2-23-23-192-197 cloudera-scm-agent]#

    Any help will be greatly appreciated.
    Thanks
    Sal

    On Tue, Jun 4, 2013 at 12:22 PM, Salman Ahmed wrote:

    Some more log snippets from cloudera agent. I killed all rogue agents, and
    put chkconfig for cloudera agent


    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory: /var/run/cloudera-scm-agent/process
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory:
    /var/run/cloudera-scm-agent/supervisor/include
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO Failed
    to connect to previous supervisor.
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread agent INFO Trying
    to connect to newly launched supervisor (Attempt 1)
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread agent INFO
    Successfully connected to supervisor
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread _cplogging INFO
    [04/Jun/2013:15:17:27] ENGINE Bus STARTING
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread _cplogging INFO
    [04/Jun/2013:15:17:27] ENGINE Started monitor thread '_TimeoutMonitor'.
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    ERROR [04/Jun/2013:15:17:27] ENGINE Error in HTTP server: shutting down
    Traceback (most recent call last):
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/process/servers.py",
    line 187, in _start_http_thread
    self.httpserver.start()
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/wsgiserver/wsgiserver2.py",
    line 1825, in start
    raise socket.error(msg)

    error: No socket could be created on ('
    ec2-23-23-192-197.compute-1.amazonaws.com', 9000) -- [Errno 99] Cannot
    assign requested address


    [root@ec2-23-23-192-197 cloudera-scm-agent]# cat cloudera-scm-agent.log
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO No
    command line vars
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Agent
    starting as pid 1511 user root(0) group root(0).
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory: /var/run/cloudera-scm-agent
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread downloader INFO
    Downloader path: /opt/cloudera/parcel-cache
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread parcel_cache INFO Using
    /opt/cloudera/parcel-cache for parcel cache
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory:
    /var/run/cloudera-scm-agent/cg
    roups
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread cgroups INFO Reusing
    /var/run/cloudera-scm-agent/cgroups/memory
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread cgroups INFO Reusing
    /var/run/cloudera-scm-agent/cgroups/cpu
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread cgroups INFO Reusing
    /var/run/cloudera-scm-agent/cgroups/blkio
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Found
    cgroups capabilities: {'has_memory': True,
    'default_memor
    y_limit_in_bytes': 9223372036854775807, 'writable_cgroup_dot_procs': True,
    'has_cpu': True, 'default_blkio_weight': 1000,
    'default _cpu_shares': 1024,
    'has_blkio': True}
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Setting
    up supervisord event monitor.
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread filesystem_map INFO
    Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs']
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread filesystem_map INFO Using
    timeout of 2.000000
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread filesystem_map INFO Using
    join timeout of 0.100000
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread filesystem_map INFO Using
    tolerance of 60.000000
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Using
    metrics_url_timeout_seconds of 30.000000
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Using
    task_metrics_timeout_seconds of 5.000000
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Using
    max_collection_wait_seconds of 10.000000
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread metrics INFO
    Importing tasktracker metric schema from file
    /usr/lib64/cmf/ag
    ent/src/cmf/monitor/tasktracker/schema.json
    [04/Jun/2013 15:17:24 +0000] 1511 MainThread __init__ INFO
    Importing metric schema from file
    /usr/lib64/cmf/agent/src/cmf/
    monitor/schema.json
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Supervised processes will inherit the following environment:
    {' CMF_CONF_DIR':
    '/etc/cloudera-scm-agent', 'CMF_AGENT_HIVE_DEFAULT_XML':
    '/etc/hive/conf.dist/hive-default.xml',
    'CMF_AGENT_MGMT_HO ME':
    '/usr/share/cmf', 'CMF_AGENT_CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs',
    'PATH': '/sbin:/usr/sbin:/bin:/usr/bin',
    'CMF_AGENT_ CDH_FLUME_HOME':
    '/usr/lib/flume-ng', 'CMF_AGENT_CDH_HBASE_HOME': '/usr/lib/hbase', 'LANG':
    'en_US.UTF-8', 'TERM': 'xterm',
    'CLOUD
    ERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar',
    'CMF_AGENT_CDH_HUE_HOME': '/usr/share/hue', 'SHLVL':
    '1',
    'CMF_AGENT_TOMCAT_HOME': '/usr/lib/bigtop-tomcat',
    'CLOUDERA_POSTGRESQL_JDBC_JAR':
    '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.ja
    r', 'CMF_AGENT_CDH_PIG_HOME': '/usr/lib/pig', 'CMF_AGENT_CDH_HIVE_HOME':
    '/usr/lib/hive', 'CLOUDERA_MYSQL_CONNECTOR_JAR':
    '/usr/sh
    are/java/mysql-connector-java.jar', 'CMF_AGENT_CDH_YARN_HOME':
    '/usr/lib/hadoop-yarn', 'CMF_AGENT_CDH_HADOOP_BIN':
    '/usr/bin/hadoo p',
    'CMF_AGENT_CDH_OOZIE_HOME': '/usr/lib/oozie',
    'CMF_AGENT_CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper',
    'CMF_AGENT_CDH_HUE_PLUGINS
    _HOME': '/usr/lib/hadoop', 'CMF_AGENT_JSVC_HOME':
    '/usr/libexec/bigtop-utils', 'CMF_AGENT_CDH_HDFS_HOME':
    '/usr/lib/hadoop-hdfs',
    'CMF_AGENT_CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce',
    'CMF_AGENT_CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'PWD': '/',
    'CM F_AGENT_CDH_HADOOP_HOME':
    '/usr/lib/hadoop', 'CMF_AGENT_CDH_IMPALA_HOME': '/usr/lib/impala'}
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Supervised processes will add the following their environment
    e nvironment:
    {'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'CMF_PACKAGE_DIR':
    '/usr/lib64/cmf/service', 'CDH_HADOOP_BIN':
    '/usr/bin /hadoop',
    'MGMT_HOME': '/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala',
    'CDH_YARN_HOME': '/usr/lib/hadoop-yarn',
    'CDH_HDFS_ HOME':
    '/usr/lib/hadoop-hdfs', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop',
    'CM_STATUS_CODES': u'STATUS_NONE
    HDFS_DFS_DIR_NOT_EMPTY' ,
    'CLOUDERA_ORACLE_CONNECTOR_JAR':
    '/usr/share/java/oracle-connector-java.jar', 'CDH_HIVE_HOME':
    '/usr/lib/hive', 'JSVC_HOME':
    '/u sr/libexec/bigtop-utils',
    'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml',
    'CLOUDERA_POSTGRESQL_JDBC_JAR':
    '/usr/share/
    cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_HBASE_HOME': '/usr/lib/hbase',
    'CDH_OOZIE_HOME': '/usr/lib/oozie',
    'CDH_ZOOKEEPER_HOME ':
    '/usr/lib/zookeeper', 'CDH_HUE_HOME': '/usr/share/hue',
    'CLOUDERA_MYSQL_CONNECTOR_JAR':
    '/usr/share/java/mysql-connector-java.j
    ar', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_PIG_HOME':
    '/usr/lib/pig', 'CDH_HTTPFS_HOME':
    '/usr/lib/hadoop-httpfs' ,
    'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'TOMCAT_HOME':
    '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'}
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory:
    /var/run/cloudera-scm-agent/pr
    ocess
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory:
    /var/run/cloudera-scm-agent/su
    pervisor
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory:
    /var/run/cloudera-scm-agent/su
    pervisor/include
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO Failed
    to connect to previous supervisor.
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread agent INFO Trying
    to connect to newly launched supervisor (Attempt 1)
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread agent INFO
    Successfully connected to supervisor
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread _cplogging INFO
    [04/Jun/2013:15:17:27] ENGINE Bus STARTING
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread _cplogging INFO
    [04/Jun/2013:15:17:27] ENGINE Started monitor thread
    '_TimeoutM onitor'.
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    ERROR [04/Jun/2013:15:17:27] ENGINE Error in HTTP server:
    sh utting down
    Traceback (most recent call last):
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/process/servers.py",
    line 187 , in
    _start_http_thread
    self.httpserver.start()
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/wsgiserver/wsgiserver2.py",
    l ine 1825, in start
    raise socket.error(msg)

    error: No socket could be created on ('
    ec2-23-23-192-197.compute-1.amazonaws.com', 9000) -- [Errno 99] Cannot
    assign requested add ress

    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    INFO [04/Jun/2013:15:17:27] ENGINE Bus STOPPING
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    INFO [04/Jun/2013:15:17:27] ENGINE HTTP Server
    cherrypy._cp
    wsgi_server.CPWSGIServer(('ec2-23-23-192-197.compute-1.amazonaws.com',
    9000)) already shut down
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    INFO [04/Jun/2013:15:17:27] ENGINE Stopped thread
    '_Timeout Monitor'.
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    INFO [04/Jun/2013:15:17:27] ENGINE Bus STOPPED
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    INFO [04/Jun/2013:15:17:27] ENGINE Bus EXITING
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    INFO [04/Jun/2013:15:17:27] ENGINE Bus EXITED
    [root@ec2-23-23-192-197 cloudera-scm-agent]# cat cloudera-scm-agent.out
    /usr/lib64/cmf/agent/src/cmf/agent.py:31: DeprecationWarning: the sha
    module is deprecated; use the hashlib module instead
    import sha
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO SCM
    Agent Version: 4.5.3
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Missing
    database jar: /usr/share/java/mysql-connector-java.jar (normal, if you're
    not using this database type)
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Missing
    database jar: /usr/share/java/oracle-connector-java.jar (normal, if you're
    not using this database type)
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Found
    database jar: /usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Using
    directory: /var/run/cloudera-scm-agent
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Using
    supervisor binary path:
    /usr/lib64/cmf/agent/src/cmf/../../build/env/bin/supervisord
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Adding
    env vars that start with CMF_AGENT_
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: HIVE_DEFAULT_XML:
    /etc/hive/conf.dist/hive-default.xml
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: MGMT_HOME: /usr/share/cmf
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HTTPFS_HOME:
    /usr/lib/hadoop-httpfs
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_FLUME_HOME: /usr/lib/flume-ng
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HBASE_HOME: /usr/lib/hbase
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HUE_HOME: /usr/share/hue
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: TOMCAT_HOME:
    /usr/lib/bigtop-tomcat
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_PIG_HOME: /usr/lib/pig
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HIVE_HOME: /usr/lib/hive
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_YARN_HOME:
    /usr/lib/hadoop-yarn
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HADOOP_BIN: /usr/bin/hadoop
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_OOZIE_HOME: /usr/lib/oozie
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_ZOOKEEPER_HOME:
    /usr/lib/zookeeper
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HUE_PLUGINS_HOME:
    /usr/lib/hadoop
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: JSVC_HOME:
    /usr/libexec/bigtop-utils
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HDFS_HOME:
    /usr/lib/hadoop-hdfs
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_MR1_HOME:
    /usr/lib/hadoop-0.20-mapreduce
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_MR2_HOME:
    /usr/lib/hadoop-mapreduce
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HADOOP_HOME: /usr/lib/hadoop
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_IMPALA_HOME: /usr/lib/impala
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Logging
    to /var/log/cloudera-scm-agent/cloudera-scm-agent.log
    Error: could not find config file
    /var/run/cloudera-scm-agent/supervisor/supervisord.conf
    For help, use /usr/lib64/cmf/agent/src/cmf/agent.py -h




    On Tue, Jun 4, 2013 at 11:02 AM, Salman Ahmed wrote:

    Just checked netstat -an | grep -i 9000, nothing is running.
    Thanks

    On Tuesday, June 4, 2013 10:05:31 AM UTC-7, Darren Lo wrote:

    Hi Salman,

    From the error, it sounds like you have a port conflict. Is something
    else listening on port 9000?

    Thanks,
    Darren

    On Tue, Jun 4, 2013 at 7:04 AM, Salman Ahmed wrote:

    Hi Guys-

    I am setting up a HBase cluster on Amazon EC2 that is reachable
    externally from the internet, so I can run jars directly from eclipse on my
    development machine.

    I have SCM server installed on Amazon EC2 cloud machine, with an
    elastic IP of 23.23.192.197

    Hostname:

    [root@54 cloudera-scm-server]# vi /etc/hosts
    [root@54 cloudera-scm-server]# hostname ec2-23-23-192-197.compute-1.**
    amazonaws.com <http://ec2-23-23-192-197.compute-1.amazonaws.com>

    Machine can ping its external IP and resolves correctly:

    [root@54 cloudera-scm-server]# ping ec2-23-23-192-197.compute-1.**
    amazonaws.com <http://ec2-23-23-192-197.compute-1.amazonaws.com>
    PING ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>(23.23.192.197) 56(84) bytes of data.
    64 bytes from ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>(23.23.192.197): icmp_seq=1 ttl=51 time=1.03 ms
    64 bytes from ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>(23.23.192.197): icmp_seq=2 ttl=51 time=0.976 ms
    64 bytes from ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>(23.23.192.197): icmp_seq=3 ttl=51 time=1.09 ms
    64 bytes from ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>(23.23.192.197): icmp_seq=4 ttl=51 time=0.936 ms
    ^C
    --- ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>ping statistics ---
    4 packets transmitted, 4 received, 0% packet loss, time 3154ms
    rtt min/avg/max/mdev = 0.936/1.010/1.097/0.060 ms

    I didn't use EC2 Cloud and rather use the classical hosts wizard. I
    edited the cloudera agent config.ini to use both the external IP and DNS
    for agent to heartbeat to cloudera manager, however the agent is unable to
    reach into the server.

    [root@54 cloudera-scm-server]# vi /etc/cloudera-scm-agent/
    [root@54 cloudera-scm-server]# vi /etc/cloudera-scm-agent/**config.ini
    [root@54 cloudera-scm-server]# service cloudera-scm-agent start
    Starting cloudera-scm-agent: [FAILED]
    [root@54 cloudera-scm-server]#
    The log is indicating: error: No socket could be created on ('
    ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>',
    9000) -- [Errno 99] Cannot assign requested address

    I have opened all ports from outside to connect on external IP but the
    issue persists,

    How do i get past this warning? The goal is to install via the
    classical SCM wizard for the cluster to be open to the internet. I was able
    to install the same cluster with local IP on a separate node, which is not
    reachable from internet, however when i run the code from eclipse from dev
    machine coming over internet, it fails to resolve HMaster and ZOOK, since
    they are on internal IP.

    Any help from users or Cloudera/BC Wong would be highly appreciated.
    Thanks
    Sal


    --
    Thanks,
    Darren
  • Philip Langdale at Jun 4, 2013 at 8:23 pm
    Salman,

    You cannot tell the Agent to do that. What you are doing with those entries
    is controlling what interface the Agent binds to when running - so it can
    only be a hostname/ip that is actually configured on the instance - but, as
    you may already know, ec2 public hostnames/ips are not configured within
    the instance, and are instead handled manually in the networking
    infrastructure, so it's impossible to do what you're trying to do.

    Now, the big question is why do you want the agent to act this way? In the
    just-release 4.6 version of CM, we now explicitly report public hostname/IP
    separately from the agent (which we query using ec2 API calls). So, the
    server will know what the public info is, and will use it where appropriate
    when generating things like web links in the UI.

    However, even without that, I don't see what advantage you will derive from
    making the agent report with public addresses. The cluster is all inside
    ec2, and will use internal addresses to communicate with itself.


    --phil

    On 4 June 2013 13:11, Salman Ahmed wrote:

    So when i put the following mapping in /etc/hosts, due to last line
    cloudera agent doesnt start->
    [root@ec2-23-23-192-197 cloudera-scm-agent]# cat /etc/hosts
    127.0.0.1 localhost.localdomain localhost
    ::1 localhost6.localdomain6 localhost6
    23.23.192.197 ec2-23-23-192-197.compute-1.amazonaws.com


    Nslookup is revealing some interesting quirks. We may need to map DNS and
    hosts resolution that is IP 23.23.192.197 always resolves to
    ec2-23-23-192-197.compute-1.amazonaws.com AND
    ec2-23-23-192-197.compute-1.amazonaws.com always resolve to 23.23.192.197
    [root@ec2-23-23-192-197 cloudera-scm-agent]# nslookup 23.23.192.197
    Server: 172.16.0.23
    Address: 172.16.0.23#53

    Non-authoritative answer:
    197.192.23.23.in-addr.arpa
    name = ec2-23-23-192-197.compute-1.amazonaws.com.

    Authoritative answers can be found from:

    [root@ec2-23-23-192-197 cloudera-scm-agent]# nslookup
    ec2-23-23-192-197.compute-1.amazonaws.com
    Server: 172.16.0.23
    Address: 172.16.0.23#53

    Non-authoritative answer:
    Name: ec2-23-23-192-197.compute-1.amazonaws.com
    Address: 10.145.232.83 <---- This should resolve to 23.23.192.197???

    [root@ec2-23-23-192-197 cloudera-scm-agent]#

    Any help will be greatly appreciated.
    Thanks
    Sal

    On Tue, Jun 4, 2013 at 12:22 PM, Salman Ahmed wrote:

    Some more log snippets from cloudera agent. I killed all rogue agents,
    and put chkconfig for cloudera agent


    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory: /var/run/cloudera-scm-agent/process
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory:
    /var/run/cloudera-scm-agent/supervisor/include
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO Failed
    to connect to previous supervisor.
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread agent INFO Trying
    to connect to newly launched supervisor (Attempt 1)
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread agent INFO
    Successfully connected to supervisor
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread _cplogging INFO
    [04/Jun/2013:15:17:27] ENGINE Bus STARTING
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread _cplogging INFO
    [04/Jun/2013:15:17:27] ENGINE Started monitor thread '_TimeoutMonitor'.
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    ERROR [04/Jun/2013:15:17:27] ENGINE Error in HTTP server: shutting down
    Traceback (most recent call last):
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/process/servers.py",
    line 187, in _start_http_thread
    self.httpserver.start()
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/wsgiserver/wsgiserver2.py",
    line 1825, in start
    raise socket.error(msg)

    error: No socket could be created on ('
    ec2-23-23-192-197.compute-1.amazonaws.com', 9000) -- [Errno 99] Cannot
    assign requested address


    [root@ec2-23-23-192-197 cloudera-scm-agent]# cat cloudera-scm-agent.log
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO No
    command line vars
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Agent
    starting as pid 1511 user root(0) group root(0).
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory: /var/run/cloudera-scm-agent
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread downloader INFO
    Downloader path: /opt/cloudera/parcel-cache
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread parcel_cache INFO Using
    /opt/cloudera/parcel-cache for parcel cache
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory:
    /var/run/cloudera-scm-agent/cg
    roups
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread cgroups INFO
    Reusing /var/run/cloudera-scm-agent/cgroups/memory
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread cgroups INFO
    Reusing /var/run/cloudera-scm-agent/cgroups/cpu
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread cgroups INFO
    Reusing /var/run/cloudera-scm-agent/cgroups/blkio
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Found
    cgroups capabilities: {'has_memory': True,
    'default_memor
    y_limit_in_bytes': 9223372036854775807, 'writable_cgroup_dot_procs': True,
    'has_cpu': True, 'default_blkio_weight': 1000,
    'default _cpu_shares': 1024,
    'has_blkio': True}
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Setting up supervisord event monitor.
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread filesystem_map INFO
    Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs']
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread filesystem_map INFO
    Using timeout of 2.000000
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread filesystem_map INFO
    Using join timeout of 0.100000
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread filesystem_map INFO
    Using tolerance of 60.000000
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Using
    metrics_url_timeout_seconds of 30.000000
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Using
    task_metrics_timeout_seconds of 5.000000
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Using
    max_collection_wait_seconds of 10.000000
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread metrics INFO
    Importing tasktracker metric schema from file
    /usr/lib64/cmf/ag
    ent/src/cmf/monitor/tasktracker/schema.json
    [04/Jun/2013 15:17:24 +0000] 1511 MainThread __init__ INFO
    Importing metric schema from file
    /usr/lib64/cmf/agent/src/cmf/
    monitor/schema.json
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Supervised processes will inherit the following environment:
    {' CMF_CONF_DIR':
    '/etc/cloudera-scm-agent', 'CMF_AGENT_HIVE_DEFAULT_XML':
    '/etc/hive/conf.dist/hive-default.xml',
    'CMF_AGENT_MGMT_HO ME':
    '/usr/share/cmf', 'CMF_AGENT_CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs',
    'PATH': '/sbin:/usr/sbin:/bin:/usr/bin',
    'CMF_AGENT_ CDH_FLUME_HOME':
    '/usr/lib/flume-ng', 'CMF_AGENT_CDH_HBASE_HOME': '/usr/lib/hbase', 'LANG':
    'en_US.UTF-8', 'TERM': 'xterm',
    'CLOUD
    ERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar',
    'CMF_AGENT_CDH_HUE_HOME': '/usr/share/hue', 'SHLVL':
    '1',
    'CMF_AGENT_TOMCAT_HOME': '/usr/lib/bigtop-tomcat',
    'CLOUDERA_POSTGRESQL_JDBC_JAR':
    '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.ja
    r', 'CMF_AGENT_CDH_PIG_HOME': '/usr/lib/pig', 'CMF_AGENT_CDH_HIVE_HOME':
    '/usr/lib/hive', 'CLOUDERA_MYSQL_CONNECTOR_JAR':
    '/usr/sh
    are/java/mysql-connector-java.jar', 'CMF_AGENT_CDH_YARN_HOME':
    '/usr/lib/hadoop-yarn', 'CMF_AGENT_CDH_HADOOP_BIN':
    '/usr/bin/hadoo p',
    'CMF_AGENT_CDH_OOZIE_HOME': '/usr/lib/oozie',
    'CMF_AGENT_CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper',
    'CMF_AGENT_CDH_HUE_PLUGINS
    _HOME': '/usr/lib/hadoop', 'CMF_AGENT_JSVC_HOME':
    '/usr/libexec/bigtop-utils', 'CMF_AGENT_CDH_HDFS_HOME':
    '/usr/lib/hadoop-hdfs',
    'CMF_AGENT_CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce',
    'CMF_AGENT_CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'PWD': '/',
    'CM F_AGENT_CDH_HADOOP_HOME':
    '/usr/lib/hadoop', 'CMF_AGENT_CDH_IMPALA_HOME': '/usr/lib/impala'}
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Supervised processes will add the following their environment
    e nvironment:
    {'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'CMF_PACKAGE_DIR':
    '/usr/lib64/cmf/service', 'CDH_HADOOP_BIN':
    '/usr/bin /hadoop',
    'MGMT_HOME': '/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala',
    'CDH_YARN_HOME': '/usr/lib/hadoop-yarn',
    'CDH_HDFS_ HOME':
    '/usr/lib/hadoop-hdfs', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop',
    'CM_STATUS_CODES': u'STATUS_NONE
    HDFS_DFS_DIR_NOT_EMPTY' ,
    'CLOUDERA_ORACLE_CONNECTOR_JAR':
    '/usr/share/java/oracle-connector-java.jar', 'CDH_HIVE_HOME':
    '/usr/lib/hive', 'JSVC_HOME':
    '/u sr/libexec/bigtop-utils',
    'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml',
    'CLOUDERA_POSTGRESQL_JDBC_JAR':
    '/usr/share/
    cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_HBASE_HOME': '/usr/lib/hbase',
    'CDH_OOZIE_HOME': '/usr/lib/oozie',
    'CDH_ZOOKEEPER_HOME ':
    '/usr/lib/zookeeper', 'CDH_HUE_HOME': '/usr/share/hue',
    'CLOUDERA_MYSQL_CONNECTOR_JAR':
    '/usr/share/java/mysql-connector-java.j
    ar', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_PIG_HOME':
    '/usr/lib/pig', 'CDH_HTTPFS_HOME':
    '/usr/lib/hadoop-httpfs' ,
    'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'TOMCAT_HOME':
    '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'}
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory:
    /var/run/cloudera-scm-agent/pr
    ocess
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory:
    /var/run/cloudera-scm-agent/su
    pervisor
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory:
    /var/run/cloudera-scm-agent/su
    pervisor/include
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO Failed
    to connect to previous supervisor.
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread agent INFO Trying
    to connect to newly launched supervisor (Attempt 1)
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread agent INFO
    Successfully connected to supervisor
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread _cplogging INFO
    [04/Jun/2013:15:17:27] ENGINE Bus STARTING
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread _cplogging INFO
    [04/Jun/2013:15:17:27] ENGINE Started monitor thread
    '_TimeoutM onitor'.
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    ERROR [04/Jun/2013:15:17:27] ENGINE Error in HTTP server:
    sh utting down
    Traceback (most recent call last):
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/process/servers.py",
    line 187 , in
    _start_http_thread
    self.httpserver.start()
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/wsgiserver/wsgiserver2.py",
    l ine 1825, in start
    raise socket.error(msg)

    error: No socket could be created on ('
    ec2-23-23-192-197.compute-1.amazonaws.com', 9000) -- [Errno 99] Cannot
    assign requested add ress

    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    INFO [04/Jun/2013:15:17:27] ENGINE Bus STOPPING
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    INFO [04/Jun/2013:15:17:27] ENGINE HTTP Server
    cherrypy._cp
    wsgi_server.CPWSGIServer(('ec2-23-23-192-197.compute-1.amazonaws.com',
    9000)) already shut down
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    INFO [04/Jun/2013:15:17:27] ENGINE Stopped thread
    '_Timeout Monitor'.
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    INFO [04/Jun/2013:15:17:27] ENGINE Bus STOPPED
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    INFO [04/Jun/2013:15:17:27] ENGINE Bus EXITING
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    INFO [04/Jun/2013:15:17:27] ENGINE Bus EXITED
    [root@ec2-23-23-192-197 cloudera-scm-agent]# cat cloudera-scm-agent.out
    /usr/lib64/cmf/agent/src/cmf/agent.py:31: DeprecationWarning: the sha
    module is deprecated; use the hashlib module instead
    import sha
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO SCM
    Agent Version: 4.5.3
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Missing database jar: /usr/share/java/mysql-connector-java.jar (normal, if
    you're not using this database type)
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Missing database jar: /usr/share/java/oracle-connector-java.jar (normal, if
    you're not using this database type)
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Found
    database jar: /usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Using
    directory: /var/run/cloudera-scm-agent
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Using
    supervisor binary path:
    /usr/lib64/cmf/agent/src/cmf/../../build/env/bin/supervisord
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO Adding
    env vars that start with CMF_AGENT_
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: HIVE_DEFAULT_XML:
    /etc/hive/conf.dist/hive-default.xml
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: MGMT_HOME: /usr/share/cmf
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HTTPFS_HOME:
    /usr/lib/hadoop-httpfs
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_FLUME_HOME: /usr/lib/flume-ng
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HBASE_HOME: /usr/lib/hbase
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HUE_HOME: /usr/share/hue
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: TOMCAT_HOME:
    /usr/lib/bigtop-tomcat
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_PIG_HOME: /usr/lib/pig
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HIVE_HOME: /usr/lib/hive
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_YARN_HOME:
    /usr/lib/hadoop-yarn
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HADOOP_BIN: /usr/bin/hadoop
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_OOZIE_HOME: /usr/lib/oozie
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_ZOOKEEPER_HOME:
    /usr/lib/zookeeper
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HUE_PLUGINS_HOME:
    /usr/lib/hadoop
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: JSVC_HOME:
    /usr/libexec/bigtop-utils
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HDFS_HOME:
    /usr/lib/hadoop-hdfs
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_MR1_HOME:
    /usr/lib/hadoop-0.20-mapreduce
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_MR2_HOME:
    /usr/lib/hadoop-mapreduce
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HADOOP_HOME: /usr/lib/hadoop
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_IMPALA_HOME: /usr/lib/impala
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Logging to /var/log/cloudera-scm-agent/cloudera-scm-agent.log
    Error: could not find config file
    /var/run/cloudera-scm-agent/supervisor/supervisord.conf
    For help, use /usr/lib64/cmf/agent/src/cmf/agent.py -h




    On Tue, Jun 4, 2013 at 11:02 AM, Salman Ahmed wrote:

    Just checked netstat -an | grep -i 9000, nothing is running.
    Thanks

    On Tuesday, June 4, 2013 10:05:31 AM UTC-7, Darren Lo wrote:

    Hi Salman,

    From the error, it sounds like you have a port conflict. Is something
    else listening on port 9000?

    Thanks,
    Darren

    On Tue, Jun 4, 2013 at 7:04 AM, Salman Ahmed wrote:

    Hi Guys-

    I am setting up a HBase cluster on Amazon EC2 that is reachable
    externally from the internet, so I can run jars directly from eclipse on my
    development machine.

    I have SCM server installed on Amazon EC2 cloud machine, with an
    elastic IP of 23.23.192.197

    Hostname:

    [root@54 cloudera-scm-server]# vi /etc/hosts
    [root@54 cloudera-scm-server]# hostname ec2-23-23-192-197.compute-1.**
    amazonaws.com <http://ec2-23-23-192-197.compute-1.amazonaws.com>

    Machine can ping its external IP and resolves correctly:

    [root@54 cloudera-scm-server]# ping ec2-23-23-192-197.compute-1.**
    amazonaws.com <http://ec2-23-23-192-197.compute-1.amazonaws.com>
    PING ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>(23.23.192.197) 56(84) bytes of data.
    64 bytes from ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>(23.23.192.197): icmp_seq=1 ttl=51 time=1.03 ms
    64 bytes from ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>(23.23.192.197): icmp_seq=2 ttl=51 time=0.976 ms
    64 bytes from ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>(23.23.192.197): icmp_seq=3 ttl=51 time=1.09 ms
    64 bytes from ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>(23.23.192.197): icmp_seq=4 ttl=51 time=0.936 ms
    ^C
    --- ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>ping statistics ---
    4 packets transmitted, 4 received, 0% packet loss, time 3154ms
    rtt min/avg/max/mdev = 0.936/1.010/1.097/0.060 ms

    I didn't use EC2 Cloud and rather use the classical hosts wizard. I
    edited the cloudera agent config.ini to use both the external IP and DNS
    for agent to heartbeat to cloudera manager, however the agent is unable to
    reach into the server.

    [root@54 cloudera-scm-server]# vi /etc/cloudera-scm-agent/
    [root@54 cloudera-scm-server]# vi /etc/cloudera-scm-agent/**config.ini
    [root@54 cloudera-scm-server]# service cloudera-scm-agent start
    Starting cloudera-scm-agent: [FAILED]
    [root@54 cloudera-scm-server]#
    The log is indicating: error: No socket could be created on ('
    ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>',
    9000) -- [Errno 99] Cannot assign requested address

    I have opened all ports from outside to connect on external IP but the
    issue persists,

    How do i get past this warning? The goal is to install via the
    classical SCM wizard for the cluster to be open to the internet. I was able
    to install the same cluster with local IP on a separate node, which is not
    reachable from internet, however when i run the code from eclipse from dev
    machine coming over internet, it fails to resolve HMaster and ZOOK, since
    they are on internal IP.

    Any help from users or Cloudera/BC Wong would be highly appreciated.
    Thanks
    Sal


    --
    Thanks,
    Darren
  • Philip Langdale at Jun 4, 2013 at 8:33 pm
    Please keep scm-users cc:ed - everyone can benefit from these discussions.

    It's hard to do what you're trying to do. ZK/HBase can't really be
    configured to do what you want. Ultimately, your options here are rather
    limited. You could use an EC2 VPC, which is probably not practical for your
    situation, or something like the Whirr Hadoop Proxy, which I don't have any
    detailed knowledge about, and I'm really not sure how to use it if you
    didn't use whirr to set up your cluster in the first place.

    The simplest solution is certainly to copy your jars into ec2. For HDFS or
    MR, you could modify the client configuration to use public hostnames, but
    with HBase you have the ZK problem you've described.

    --phil

    On 4 June 2013 13:28, Salman Ahmed wrote:

    Hi Philip-

    Thank you for the insight. What I was attempting was to deploy the cluster
    on external public IP, with all processes reachable from internet, so from
    eclipse running connected to home WIFI, code can directly execute hbase
    tables. Presently it times out, as the ips returned by ZOOK to client are
    internal and thus socket/connection fails.

    Any advice how to make the cluster available for direct connect from home
    machine to execute code for development purpose, with out having to copy
    jar to the EC2 machine itself to run.

    Best Regards
    Sal

    On Tue, Jun 4, 2013 at 1:23 PM, Philip Langdale wrote:

    Salman,

    You cannot tell the Agent to do that. What you are doing with those
    entries is controlling what interface the Agent binds to when running - so
    it can only be a hostname/ip that is actually configured on the instance -
    but, as you may already know, ec2 public hostnames/ips are not configured
    within the instance, and are instead handled manually in the networking
    infrastructure, so it's impossible to do what you're trying to do.

    Now, the big question is why do you want the agent to act this way? In
    the just-release 4.6 version of CM, we now explicitly report public
    hostname/IP separately from the agent (which we query using ec2 API calls).
    So, the server will know what the public info is, and will use it where
    appropriate when generating things like web links in the UI.

    However, even without that, I don't see what advantage you will derive
    from making the agent report with public addresses. The cluster is all
    inside ec2, and will use internal addresses to communicate with itself.


    --phil

    On 4 June 2013 13:11, Salman Ahmed wrote:

    So when i put the following mapping in /etc/hosts, due to last line
    cloudera agent doesnt start->
    [root@ec2-23-23-192-197 cloudera-scm-agent]# cat /etc/hosts
    127.0.0.1 localhost.localdomain localhost
    ::1 localhost6.localdomain6 localhost6
    23.23.192.197 ec2-23-23-192-197.compute-1.amazonaws.com


    Nslookup is revealing some interesting quirks. We may need to map DNS
    and hosts resolution that is IP 23.23.192.197 always resolves to
    ec2-23-23-192-197.compute-1.amazonaws.com AND
    ec2-23-23-192-197.compute-1.amazonaws.com always resolve to
    23.23.192.197
    [root@ec2-23-23-192-197 cloudera-scm-agent]# nslookup 23.23.192.197
    Server: 172.16.0.23
    Address: 172.16.0.23#53

    Non-authoritative answer:
    197.192.23.23.in-addr.arpa
    name = ec2-23-23-192-197.compute-1.amazonaws.com.

    Authoritative answers can be found from:

    [root@ec2-23-23-192-197 cloudera-scm-agent]# nslookup
    ec2-23-23-192-197.compute-1.amazonaws.com
    Server: 172.16.0.23
    Address: 172.16.0.23#53

    Non-authoritative answer:
    Name: ec2-23-23-192-197.compute-1.amazonaws.com
    Address: 10.145.232.83 <---- This should resolve to 23.23.192.197???

    [root@ec2-23-23-192-197 cloudera-scm-agent]#

    Any help will be greatly appreciated.
    Thanks
    Sal

    On Tue, Jun 4, 2013 at 12:22 PM, Salman Ahmed wrote:

    Some more log snippets from cloudera agent. I killed all rogue agents,
    and put chkconfig for cloudera agent


    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory: /var/run/cloudera-scm-agent/process
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory:
    /var/run/cloudera-scm-agent/supervisor/include
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Failed to connect to previous supervisor.
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread agent INFO
    Trying to connect to newly launched supervisor (Attempt 1)
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread agent INFO
    Successfully connected to supervisor
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread _cplogging INFO
    [04/Jun/2013:15:17:27] ENGINE Bus STARTING
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread _cplogging INFO
    [04/Jun/2013:15:17:27] ENGINE Started monitor thread '_TimeoutMonitor'.
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    ERROR [04/Jun/2013:15:17:27] ENGINE Error in HTTP server: shutting down
    Traceback (most recent call last):
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/process/servers.py",
    line 187, in _start_http_thread
    self.httpserver.start()
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/wsgiserver/wsgiserver2.py",
    line 1825, in start
    raise socket.error(msg)

    error: No socket could be created on ('
    ec2-23-23-192-197.compute-1.amazonaws.com', 9000) -- [Errno 99] Cannot
    assign requested address


    [root@ec2-23-23-192-197 cloudera-scm-agent]# cat cloudera-scm-agent.log
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO No
    command line vars
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Agent starting as pid 1511 user root(0) group root(0).
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory: /var/run/cloudera-scm-agent
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread downloader INFO
    Downloader path: /opt/cloudera/parcel-cache
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread parcel_cache INFO
    Using /opt/cloudera/parcel-cache for parcel cache
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory:
    /var/run/cloudera-scm-agent/cg
    roups
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread cgroups INFO
    Reusing /var/run/cloudera-scm-agent/cgroups/memory
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread cgroups INFO
    Reusing /var/run/cloudera-scm-agent/cgroups/cpu
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread cgroups INFO
    Reusing /var/run/cloudera-scm-agent/cgroups/blkio
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Found cgroups capabilities: {'has_memory': True,
    'default_memor
    y_limit_in_bytes': 9223372036854775807, 'writable_cgroup_dot_procs': True,
    'has_cpu': True, 'default_blkio_weight': 1000,
    'default _cpu_shares': 1024,
    'has_blkio': True}
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Setting up supervisord event monitor.
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread filesystem_map INFO
    Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs']
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread filesystem_map INFO
    Using timeout of 2.000000
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread filesystem_map INFO
    Using join timeout of 0.100000
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread filesystem_map INFO
    Using tolerance of 60.000000
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Using metrics_url_timeout_seconds of 30.000000
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Using task_metrics_timeout_seconds of 5.000000
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Using max_collection_wait_seconds of 10.000000
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread metrics INFO
    Importing tasktracker metric schema from file
    /usr/lib64/cmf/ag
    ent/src/cmf/monitor/tasktracker/schema.json
    [04/Jun/2013 15:17:24 +0000] 1511 MainThread __init__ INFO
    Importing metric schema from file
    /usr/lib64/cmf/agent/src/cmf/
    monitor/schema.json
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Supervised processes will inherit the following environment:
    {' CMF_CONF_DIR':
    '/etc/cloudera-scm-agent', 'CMF_AGENT_HIVE_DEFAULT_XML':
    '/etc/hive/conf.dist/hive-default.xml',
    'CMF_AGENT_MGMT_HO ME':
    '/usr/share/cmf', 'CMF_AGENT_CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs',
    'PATH': '/sbin:/usr/sbin:/bin:/usr/bin',
    'CMF_AGENT_ CDH_FLUME_HOME':
    '/usr/lib/flume-ng', 'CMF_AGENT_CDH_HBASE_HOME': '/usr/lib/hbase', 'LANG':
    'en_US.UTF-8', 'TERM': 'xterm',
    'CLOUD
    ERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar',
    'CMF_AGENT_CDH_HUE_HOME': '/usr/share/hue', 'SHLVL':
    '1',
    'CMF_AGENT_TOMCAT_HOME': '/usr/lib/bigtop-tomcat',
    'CLOUDERA_POSTGRESQL_JDBC_JAR':
    '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.ja
    r', 'CMF_AGENT_CDH_PIG_HOME': '/usr/lib/pig', 'CMF_AGENT_CDH_HIVE_HOME':
    '/usr/lib/hive', 'CLOUDERA_MYSQL_CONNECTOR_JAR':
    '/usr/sh
    are/java/mysql-connector-java.jar', 'CMF_AGENT_CDH_YARN_HOME':
    '/usr/lib/hadoop-yarn', 'CMF_AGENT_CDH_HADOOP_BIN':
    '/usr/bin/hadoo p',
    'CMF_AGENT_CDH_OOZIE_HOME': '/usr/lib/oozie',
    'CMF_AGENT_CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper',
    'CMF_AGENT_CDH_HUE_PLUGINS
    _HOME': '/usr/lib/hadoop', 'CMF_AGENT_JSVC_HOME':
    '/usr/libexec/bigtop-utils', 'CMF_AGENT_CDH_HDFS_HOME':
    '/usr/lib/hadoop-hdfs',
    'CMF_AGENT_CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce',
    'CMF_AGENT_CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'PWD': '/',
    'CM F_AGENT_CDH_HADOOP_HOME':
    '/usr/lib/hadoop', 'CMF_AGENT_CDH_IMPALA_HOME': '/usr/lib/impala'}
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Supervised processes will add the following their environment
    e nvironment:
    {'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'CMF_PACKAGE_DIR':
    '/usr/lib64/cmf/service', 'CDH_HADOOP_BIN':
    '/usr/bin /hadoop',
    'MGMT_HOME': '/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala',
    'CDH_YARN_HOME': '/usr/lib/hadoop-yarn',
    'CDH_HDFS_ HOME':
    '/usr/lib/hadoop-hdfs', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop',
    'CM_STATUS_CODES': u'STATUS_NONE
    HDFS_DFS_DIR_NOT_EMPTY' ,
    'CLOUDERA_ORACLE_CONNECTOR_JAR':
    '/usr/share/java/oracle-connector-java.jar', 'CDH_HIVE_HOME':
    '/usr/lib/hive', 'JSVC_HOME':
    '/u sr/libexec/bigtop-utils',
    'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml',
    'CLOUDERA_POSTGRESQL_JDBC_JAR':
    '/usr/share/
    cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_HBASE_HOME': '/usr/lib/hbase',
    'CDH_OOZIE_HOME': '/usr/lib/oozie',
    'CDH_ZOOKEEPER_HOME ':
    '/usr/lib/zookeeper', 'CDH_HUE_HOME': '/usr/share/hue',
    'CLOUDERA_MYSQL_CONNECTOR_JAR':
    '/usr/share/java/mysql-connector-java.j
    ar', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_PIG_HOME':
    '/usr/lib/pig', 'CDH_HTTPFS_HOME':
    '/usr/lib/hadoop-httpfs' ,
    'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'TOMCAT_HOME':
    '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'}
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory:
    /var/run/cloudera-scm-agent/pr
    ocess
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory:
    /var/run/cloudera-scm-agent/su
    pervisor
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory:
    /var/run/cloudera-scm-agent/su
    pervisor/include
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Failed to connect to previous supervisor.
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread agent INFO
    Trying to connect to newly launched supervisor (Attempt 1)
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread agent INFO
    Successfully connected to supervisor
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread _cplogging INFO
    [04/Jun/2013:15:17:27] ENGINE Bus STARTING
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread _cplogging INFO
    [04/Jun/2013:15:17:27] ENGINE Started monitor thread
    '_TimeoutM onitor'.
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    ERROR [04/Jun/2013:15:17:27] ENGINE Error in HTTP server:
    sh utting down
    Traceback (most recent call last):
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/process/servers.py",
    line 187 , in
    _start_http_thread
    self.httpserver.start()
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/wsgiserver/wsgiserver2.py",
    l ine 1825, in start
    raise socket.error(msg)

    error: No socket could be created on ('
    ec2-23-23-192-197.compute-1.amazonaws.com', 9000) -- [Errno 99] Cannot
    assign requested add ress

    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    INFO [04/Jun/2013:15:17:27] ENGINE Bus STOPPING
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    INFO [04/Jun/2013:15:17:27] ENGINE HTTP Server
    cherrypy._cp
    wsgi_server.CPWSGIServer(('ec2-23-23-192-197.compute-1.amazonaws.com',
    9000)) already shut down
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    INFO [04/Jun/2013:15:17:27] ENGINE Stopped thread
    '_Timeout Monitor'.
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    INFO [04/Jun/2013:15:17:27] ENGINE Bus STOPPED
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    INFO [04/Jun/2013:15:17:27] ENGINE Bus EXITING
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    INFO [04/Jun/2013:15:17:27] ENGINE Bus EXITED
    [root@ec2-23-23-192-197 cloudera-scm-agent]# cat cloudera-scm-agent.out
    /usr/lib64/cmf/agent/src/cmf/agent.py:31: DeprecationWarning: the sha
    module is deprecated; use the hashlib module instead
    import sha
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO SCM
    Agent Version: 4.5.3
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Missing database jar: /usr/share/java/mysql-connector-java.jar (normal, if
    you're not using this database type)
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Missing database jar: /usr/share/java/oracle-connector-java.jar (normal, if
    you're not using this database type)
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Found database jar: /usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Using directory: /var/run/cloudera-scm-agent
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Using supervisor binary path:
    /usr/lib64/cmf/agent/src/cmf/../../build/env/bin/supervisord
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Adding env vars that start with CMF_AGENT_
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: HIVE_DEFAULT_XML:
    /etc/hive/conf.dist/hive-default.xml
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: MGMT_HOME: /usr/share/cmf
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HTTPFS_HOME:
    /usr/lib/hadoop-httpfs
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_FLUME_HOME: /usr/lib/flume-ng
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HBASE_HOME: /usr/lib/hbase
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HUE_HOME: /usr/share/hue
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: TOMCAT_HOME:
    /usr/lib/bigtop-tomcat
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_PIG_HOME: /usr/lib/pig
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HIVE_HOME: /usr/lib/hive
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_YARN_HOME:
    /usr/lib/hadoop-yarn
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HADOOP_BIN: /usr/bin/hadoop
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_OOZIE_HOME: /usr/lib/oozie
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_ZOOKEEPER_HOME:
    /usr/lib/zookeeper
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HUE_PLUGINS_HOME:
    /usr/lib/hadoop
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: JSVC_HOME:
    /usr/libexec/bigtop-utils
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HDFS_HOME:
    /usr/lib/hadoop-hdfs
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_MR1_HOME:
    /usr/lib/hadoop-0.20-mapreduce
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_MR2_HOME:
    /usr/lib/hadoop-mapreduce
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HADOOP_HOME: /usr/lib/hadoop
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_IMPALA_HOME: /usr/lib/impala
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Logging to /var/log/cloudera-scm-agent/cloudera-scm-agent.log
    Error: could not find config file
    /var/run/cloudera-scm-agent/supervisor/supervisord.conf
    For help, use /usr/lib64/cmf/agent/src/cmf/agent.py -h




    On Tue, Jun 4, 2013 at 11:02 AM, Salman Ahmed wrote:

    Just checked netstat -an | grep -i 9000, nothing is running.
    Thanks

    On Tuesday, June 4, 2013 10:05:31 AM UTC-7, Darren Lo wrote:

    Hi Salman,

    From the error, it sounds like you have a port conflict. Is something
    else listening on port 9000?

    Thanks,
    Darren

    On Tue, Jun 4, 2013 at 7:04 AM, Salman Ahmed wrote:

    Hi Guys-

    I am setting up a HBase cluster on Amazon EC2 that is reachable
    externally from the internet, so I can run jars directly from eclipse on my
    development machine.

    I have SCM server installed on Amazon EC2 cloud machine, with an
    elastic IP of 23.23.192.197

    Hostname:

    [root@54 cloudera-scm-server]# vi /etc/hosts
    [root@54 cloudera-scm-server]# hostname ec2-23-23-192-197.compute-1.
    **amazonaws.com <http://ec2-23-23-192-197.compute-1.amazonaws.com>

    Machine can ping its external IP and resolves correctly:

    [root@54 cloudera-scm-server]# ping ec2-23-23-192-197.compute-1.**
    amazonaws.com <http://ec2-23-23-192-197.compute-1.amazonaws.com>
    PING ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>(23.23.192.197) 56(84) bytes of data.
    64 bytes from ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>(23.23.192.197): icmp_seq=1 ttl=51 time=1.03 ms
    64 bytes from ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>(23.23.192.197): icmp_seq=2 ttl=51 time=0.976 ms
    64 bytes from ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>(23.23.192.197): icmp_seq=3 ttl=51 time=1.09 ms
    64 bytes from ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>(23.23.192.197): icmp_seq=4 ttl=51 time=0.936 ms
    ^C
    --- ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>ping statistics ---
    4 packets transmitted, 4 received, 0% packet loss, time 3154ms
    rtt min/avg/max/mdev = 0.936/1.010/1.097/0.060 ms

    I didn't use EC2 Cloud and rather use the classical hosts wizard. I
    edited the cloudera agent config.ini to use both the external IP and DNS
    for agent to heartbeat to cloudera manager, however the agent is unable to
    reach into the server.

    [root@54 cloudera-scm-server]# vi /etc/cloudera-scm-agent/
    [root@54 cloudera-scm-server]# vi /etc/cloudera-scm-agent/**
    config.ini
    [root@54 cloudera-scm-server]# service cloudera-scm-agent start
    Starting cloudera-scm-agent: [FAILED]
    [root@54 cloudera-scm-server]#
    The log is indicating: error: No socket could be created on ('
    ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>',
    9000) -- [Errno 99] Cannot assign requested address

    I have opened all ports from outside to connect on external IP but
    the issue persists,

    How do i get past this warning? The goal is to install via the
    classical SCM wizard for the cluster to be open to the internet. I was able
    to install the same cluster with local IP on a separate node, which is not
    reachable from internet, however when i run the code from eclipse from dev
    machine coming over internet, it fails to resolve HMaster and ZOOK, since
    they are on internal IP.

    Any help from users or Cloudera/BC Wong would be highly appreciated.
    Thanks
    Sal


    --
    Thanks,
    Darren
  • Salman Ahmed at Jun 4, 2013 at 8:39 pm
    Thank you Phil. This information is very helpful.

    Kind Regards

    On Tue, Jun 4, 2013 at 1:33 PM, Philip Langdale wrote:

    Please keep scm-users cc:ed - everyone can benefit from these discussions.

    It's hard to do what you're trying to do. ZK/HBase can't really be
    configured to do what you want. Ultimately, your options here are rather
    limited. You could use an EC2 VPC, which is probably not practical for your
    situation, or something like the Whirr Hadoop Proxy, which I don't have any
    detailed knowledge about, and I'm really not sure how to use it if you
    didn't use whirr to set up your cluster in the first place.

    The simplest solution is certainly to copy your jars into ec2. For HDFS or
    MR, you could modify the client configuration to use public hostnames, but
    with HBase you have the ZK problem you've described.

    --phil

    On 4 June 2013 13:28, Salman Ahmed wrote:

    Hi Philip-

    Thank you for the insight. What I was attempting was to deploy the
    cluster on external public IP, with all processes reachable from internet,
    so from eclipse running connected to home WIFI, code can directly execute
    hbase tables. Presently it times out, as the ips returned by ZOOK to client
    are internal and thus socket/connection fails.

    Any advice how to make the cluster available for direct connect from home
    machine to execute code for development purpose, with out having to copy
    jar to the EC2 machine itself to run.

    Best Regards
    Sal

    On Tue, Jun 4, 2013 at 1:23 PM, Philip Langdale wrote:

    Salman,

    You cannot tell the Agent to do that. What you are doing with those
    entries is controlling what interface the Agent binds to when running - so
    it can only be a hostname/ip that is actually configured on the instance -
    but, as you may already know, ec2 public hostnames/ips are not configured
    within the instance, and are instead handled manually in the networking
    infrastructure, so it's impossible to do what you're trying to do.

    Now, the big question is why do you want the agent to act this way? In
    the just-release 4.6 version of CM, we now explicitly report public
    hostname/IP separately from the agent (which we query using ec2 API calls).
    So, the server will know what the public info is, and will use it where
    appropriate when generating things like web links in the UI.

    However, even without that, I don't see what advantage you will derive
    from making the agent report with public addresses. The cluster is all
    inside ec2, and will use internal addresses to communicate with itself.


    --phil

    On 4 June 2013 13:11, Salman Ahmed wrote:

    So when i put the following mapping in /etc/hosts, due to last line
    cloudera agent doesnt start->
    [root@ec2-23-23-192-197 cloudera-scm-agent]# cat /etc/hosts
    127.0.0.1 localhost.localdomain localhost
    ::1 localhost6.localdomain6 localhost6
    23.23.192.197 ec2-23-23-192-197.compute-1.amazonaws.com


    Nslookup is revealing some interesting quirks. We may need to map DNS
    and hosts resolution that is IP 23.23.192.197 always resolves to
    ec2-23-23-192-197.compute-1.amazonaws.com AND
    ec2-23-23-192-197.compute-1.amazonaws.com always resolve to
    23.23.192.197
    [root@ec2-23-23-192-197 cloudera-scm-agent]# nslookup 23.23.192.197
    Server: 172.16.0.23
    Address: 172.16.0.23#53

    Non-authoritative answer:
    197.192.23.23.in-addr.arpa
    name = ec2-23-23-192-197.compute-1.amazonaws.com.

    Authoritative answers can be found from:

    [root@ec2-23-23-192-197 cloudera-scm-agent]# nslookup
    ec2-23-23-192-197.compute-1.amazonaws.com
    Server: 172.16.0.23
    Address: 172.16.0.23#53

    Non-authoritative answer:
    Name: ec2-23-23-192-197.compute-1.amazonaws.com
    Address: 10.145.232.83 <---- This should resolve to 23.23.192.197???

    [root@ec2-23-23-192-197 cloudera-scm-agent]#

    Any help will be greatly appreciated.
    Thanks
    Sal

    On Tue, Jun 4, 2013 at 12:22 PM, Salman Ahmed wrote:

    Some more log snippets from cloudera agent. I killed all rogue agents,
    and put chkconfig for cloudera agent


    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory: /var/run/cloudera-scm-agent/process
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory:
    /var/run/cloudera-scm-agent/supervisor/include
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Failed to connect to previous supervisor.
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread agent INFO
    Trying to connect to newly launched supervisor (Attempt 1)
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread agent INFO
    Successfully connected to supervisor
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread _cplogging INFO
    [04/Jun/2013:15:17:27] ENGINE Bus STARTING
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread _cplogging INFO
    [04/Jun/2013:15:17:27] ENGINE Started monitor thread '_TimeoutMonitor'.
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    ERROR [04/Jun/2013:15:17:27] ENGINE Error in HTTP server: shutting down
    Traceback (most recent call last):
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/process/servers.py",
    line 187, in _start_http_thread
    self.httpserver.start()
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/wsgiserver/wsgiserver2.py",
    line 1825, in start
    raise socket.error(msg)

    error: No socket could be created on ('
    ec2-23-23-192-197.compute-1.amazonaws.com', 9000) -- [Errno 99]
    Cannot assign requested address


    [root@ec2-23-23-192-197 cloudera-scm-agent]# cat
    cloudera-scm-agent.log
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO No
    command line vars
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Agent starting as pid 1511 user root(0) group root(0).
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory: /var/run/cloudera-scm-agent
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread downloader INFO
    Downloader path: /opt/cloudera/parcel-cache
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread parcel_cache INFO
    Using /opt/cloudera/parcel-cache for parcel cache
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory:
    /var/run/cloudera-scm-agent/cg
    roups
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread cgroups INFO
    Reusing /var/run/cloudera-scm-agent/cgroups/memory
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread cgroups INFO
    Reusing /var/run/cloudera-scm-agent/cgroups/cpu
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread cgroups INFO
    Reusing /var/run/cloudera-scm-agent/cgroups/blkio
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Found cgroups capabilities: {'has_memory': True,
    'default_memor
    y_limit_in_bytes': 9223372036854775807, 'writable_cgroup_dot_procs': True,
    'has_cpu': True, 'default_blkio_weight': 1000,
    'default _cpu_shares': 1024,
    'has_blkio': True}
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Setting up supervisord event monitor.
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread filesystem_map INFO
    Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs']
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread filesystem_map INFO
    Using timeout of 2.000000
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread filesystem_map INFO
    Using join timeout of 0.100000
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread filesystem_map INFO
    Using tolerance of 60.000000
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Using metrics_url_timeout_seconds of 30.000000
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Using task_metrics_timeout_seconds of 5.000000
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Using max_collection_wait_seconds of 10.000000
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread metrics INFO
    Importing tasktracker metric schema from file
    /usr/lib64/cmf/ag
    ent/src/cmf/monitor/tasktracker/schema.json
    [04/Jun/2013 15:17:24 +0000] 1511 MainThread __init__ INFO
    Importing metric schema from file
    /usr/lib64/cmf/agent/src/cmf/
    monitor/schema.json
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Supervised processes will inherit the following environment:
    {' CMF_CONF_DIR':
    '/etc/cloudera-scm-agent', 'CMF_AGENT_HIVE_DEFAULT_XML':
    '/etc/hive/conf.dist/hive-default.xml',
    'CMF_AGENT_MGMT_HO ME':
    '/usr/share/cmf', 'CMF_AGENT_CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs',
    'PATH': '/sbin:/usr/sbin:/bin:/usr/bin',
    'CMF_AGENT_ CDH_FLUME_HOME':
    '/usr/lib/flume-ng', 'CMF_AGENT_CDH_HBASE_HOME': '/usr/lib/hbase', 'LANG':
    'en_US.UTF-8', 'TERM': 'xterm',
    'CLOUD
    ERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar',
    'CMF_AGENT_CDH_HUE_HOME': '/usr/share/hue', 'SHLVL':
    '1',
    'CMF_AGENT_TOMCAT_HOME': '/usr/lib/bigtop-tomcat',
    'CLOUDERA_POSTGRESQL_JDBC_JAR':
    '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.ja
    r', 'CMF_AGENT_CDH_PIG_HOME': '/usr/lib/pig', 'CMF_AGENT_CDH_HIVE_HOME':
    '/usr/lib/hive', 'CLOUDERA_MYSQL_CONNECTOR_JAR':
    '/usr/sh
    are/java/mysql-connector-java.jar', 'CMF_AGENT_CDH_YARN_HOME':
    '/usr/lib/hadoop-yarn', 'CMF_AGENT_CDH_HADOOP_BIN':
    '/usr/bin/hadoo p',
    'CMF_AGENT_CDH_OOZIE_HOME': '/usr/lib/oozie',
    'CMF_AGENT_CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper',
    'CMF_AGENT_CDH_HUE_PLUGINS
    _HOME': '/usr/lib/hadoop', 'CMF_AGENT_JSVC_HOME':
    '/usr/libexec/bigtop-utils', 'CMF_AGENT_CDH_HDFS_HOME':
    '/usr/lib/hadoop-hdfs',
    'CMF_AGENT_CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce',
    'CMF_AGENT_CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'PWD': '/',
    'CM F_AGENT_CDH_HADOOP_HOME':
    '/usr/lib/hadoop', 'CMF_AGENT_CDH_IMPALA_HOME': '/usr/lib/impala'}
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Supervised processes will add the following their environment
    e nvironment:
    {'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'CMF_PACKAGE_DIR':
    '/usr/lib64/cmf/service', 'CDH_HADOOP_BIN':
    '/usr/bin /hadoop',
    'MGMT_HOME': '/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala',
    'CDH_YARN_HOME': '/usr/lib/hadoop-yarn',
    'CDH_HDFS_ HOME':
    '/usr/lib/hadoop-hdfs', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop',
    'CM_STATUS_CODES': u'STATUS_NONE
    HDFS_DFS_DIR_NOT_EMPTY' ,
    'CLOUDERA_ORACLE_CONNECTOR_JAR':
    '/usr/share/java/oracle-connector-java.jar', 'CDH_HIVE_HOME':
    '/usr/lib/hive', 'JSVC_HOME':
    '/u sr/libexec/bigtop-utils',
    'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml',
    'CLOUDERA_POSTGRESQL_JDBC_JAR':
    '/usr/share/
    cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_HBASE_HOME': '/usr/lib/hbase',
    'CDH_OOZIE_HOME': '/usr/lib/oozie',
    'CDH_ZOOKEEPER_HOME ':
    '/usr/lib/zookeeper', 'CDH_HUE_HOME': '/usr/share/hue',
    'CLOUDERA_MYSQL_CONNECTOR_JAR':
    '/usr/share/java/mysql-connector-java.j
    ar', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_PIG_HOME':
    '/usr/lib/pig', 'CDH_HTTPFS_HOME':
    '/usr/lib/hadoop-httpfs' ,
    'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'TOMCAT_HOME':
    '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'}
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory:
    /var/run/cloudera-scm-agent/pr
    ocess
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory:
    /var/run/cloudera-scm-agent/su
    pervisor
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Re-using pre-existing directory:
    /var/run/cloudera-scm-agent/su
    pervisor/include
    [04/Jun/2013 15:17:25 +0000] 1511 MainThread agent INFO
    Failed to connect to previous supervisor.
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread agent INFO
    Trying to connect to newly launched supervisor (Attempt 1)
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread agent INFO
    Successfully connected to supervisor
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread _cplogging INFO
    [04/Jun/2013:15:17:27] ENGINE Bus STARTING
    [04/Jun/2013 15:17:27 +0000] 1511 MainThread _cplogging INFO
    [04/Jun/2013:15:17:27] ENGINE Started monitor thread
    '_TimeoutM onitor'.
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    ERROR [04/Jun/2013:15:17:27] ENGINE Error in HTTP server:
    sh utting down
    Traceback (most recent call last):
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/process/servers.py",
    line 187 , in
    _start_http_thread
    self.httpserver.start()
    File
    "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/wsgiserver/wsgiserver2.py",
    l ine 1825, in start
    raise socket.error(msg)

    error: No socket could be created on ('
    ec2-23-23-192-197.compute-1.amazonaws.com', 9000) -- [Errno 99]
    Cannot assign requested add
    ress

    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    INFO [04/Jun/2013:15:17:27] ENGINE Bus STOPPING
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    INFO [04/Jun/2013:15:17:27] ENGINE HTTP Server
    cherrypy._cp
    wsgi_server.CPWSGIServer(('ec2-23-23-192-197.compute-1.amazonaws.com',
    9000)) already shut down
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    INFO [04/Jun/2013:15:17:27] ENGINE Stopped thread
    '_Timeout Monitor'.
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    INFO [04/Jun/2013:15:17:27] ENGINE Bus STOPPED
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    INFO [04/Jun/2013:15:17:27] ENGINE Bus EXITING
    [04/Jun/2013 15:17:27 +0000] 1511 HTTPServer Thread-3 _cplogging
    INFO [04/Jun/2013:15:17:27] ENGINE Bus EXITED
    [root@ec2-23-23-192-197 cloudera-scm-agent]# cat
    cloudera-scm-agent.out
    /usr/lib64/cmf/agent/src/cmf/agent.py:31: DeprecationWarning: the sha
    module is deprecated; use the hashlib module instead
    import sha
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO SCM
    Agent Version: 4.5.3
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Missing database jar: /usr/share/java/mysql-connector-java.jar (normal, if
    you're not using this database type)
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Missing database jar: /usr/share/java/oracle-connector-java.jar (normal, if
    you're not using this database type)
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Found database jar: /usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Using directory: /var/run/cloudera-scm-agent
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Using supervisor binary path:
    /usr/lib64/cmf/agent/src/cmf/../../build/env/bin/supervisord
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Adding env vars that start with CMF_AGENT_
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: HIVE_DEFAULT_XML:
    /etc/hive/conf.dist/hive-default.xml
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: MGMT_HOME: /usr/share/cmf
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HTTPFS_HOME:
    /usr/lib/hadoop-httpfs
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_FLUME_HOME: /usr/lib/flume-ng
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HBASE_HOME: /usr/lib/hbase
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HUE_HOME: /usr/share/hue
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: TOMCAT_HOME:
    /usr/lib/bigtop-tomcat
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_PIG_HOME: /usr/lib/pig
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HIVE_HOME: /usr/lib/hive
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_YARN_HOME:
    /usr/lib/hadoop-yarn
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HADOOP_BIN: /usr/bin/hadoop
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_OOZIE_HOME: /usr/lib/oozie
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_ZOOKEEPER_HOME:
    /usr/lib/zookeeper
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HUE_PLUGINS_HOME:
    /usr/lib/hadoop
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: JSVC_HOME:
    /usr/libexec/bigtop-utils
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HDFS_HOME:
    /usr/lib/hadoop-hdfs
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_MR1_HOME:
    /usr/lib/hadoop-0.20-mapreduce
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_MR2_HOME:
    /usr/lib/hadoop-mapreduce
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_HADOOP_HOME: /usr/lib/hadoop
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    explicitly passing env var to supervisee: CDH_IMPALA_HOME: /usr/lib/impala
    [04/Jun/2013 15:17:23 +0000] 1511 MainThread agent INFO
    Logging to /var/log/cloudera-scm-agent/cloudera-scm-agent.log
    Error: could not find config file
    /var/run/cloudera-scm-agent/supervisor/supervisord.conf
    For help, use /usr/lib64/cmf/agent/src/cmf/agent.py -h




    On Tue, Jun 4, 2013 at 11:02 AM, Salman Ahmed wrote:

    Just checked netstat -an | grep -i 9000, nothing is running.
    Thanks

    On Tuesday, June 4, 2013 10:05:31 AM UTC-7, Darren Lo wrote:

    Hi Salman,

    From the error, it sounds like you have a port conflict. Is
    something else listening on port 9000?

    Thanks,
    Darren

    On Tue, Jun 4, 2013 at 7:04 AM, Salman Ahmed wrote:

    Hi Guys-

    I am setting up a HBase cluster on Amazon EC2 that is reachable
    externally from the internet, so I can run jars directly from eclipse on my
    development machine.

    I have SCM server installed on Amazon EC2 cloud machine, with an
    elastic IP of 23.23.192.197

    Hostname:

    [root@54 cloudera-scm-server]# vi /etc/hosts
    [root@54 cloudera-scm-server]# hostname
    ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>

    Machine can ping its external IP and resolves correctly:

    [root@54 cloudera-scm-server]# ping ec2-23-23-192-197.compute-1.**
    amazonaws.com <http://ec2-23-23-192-197.compute-1.amazonaws.com>
    PING ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>(23.23.192.197) 56(84) bytes of data.
    64 bytes from ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>(23.23.192.197): icmp_seq=1 ttl=51 time=1.03 ms
    64 bytes from ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>(23.23.192.197): icmp_seq=2 ttl=51 time=0.976 ms
    64 bytes from ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>(23.23.192.197): icmp_seq=3 ttl=51 time=1.09 ms
    64 bytes from ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>(23.23.192.197): icmp_seq=4 ttl=51 time=0.936 ms
    ^C
    --- ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>ping statistics ---
    4 packets transmitted, 4 received, 0% packet loss, time 3154ms
    rtt min/avg/max/mdev = 0.936/1.010/1.097/0.060 ms

    I didn't use EC2 Cloud and rather use the classical hosts wizard. I
    edited the cloudera agent config.ini to use both the external IP and DNS
    for agent to heartbeat to cloudera manager, however the agent is unable to
    reach into the server.

    [root@54 cloudera-scm-server]# vi /etc/cloudera-scm-agent/
    [root@54 cloudera-scm-server]# vi /etc/cloudera-scm-agent/**
    config.ini
    [root@54 cloudera-scm-server]# service cloudera-scm-agent start
    Starting cloudera-scm-agent: [FAILED]
    [root@54 cloudera-scm-server]#
    The log is indicating: error: No socket could be created on ('
    ec2-23-23-192-197.compute-1.**amazonaws.com<http://ec2-23-23-192-197.compute-1.amazonaws.com>',
    9000) -- [Errno 99] Cannot assign requested address

    I have opened all ports from outside to connect on external IP but
    the issue persists,

    How do i get past this warning? The goal is to install via the
    classical SCM wizard for the cluster to be open to the internet. I was able
    to install the same cluster with local IP on a separate node, which is not
    reachable from internet, however when i run the code from eclipse from dev
    machine coming over internet, it fails to resolve HMaster and ZOOK, since
    they are on internal IP.

    Any help from users or Cloudera/BC Wong would be highly appreciated.
    Thanks
    Sal


    --
    Thanks,
    Darren

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedJun 4, '13 at 2:04p
activeJun 4, '13 at 8:39p
posts8
users3
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase