Role been out of contact with CM for too long

Hi Rex,

Is the cloudera manager agent running on that host?
service cloudera-scm-agent status

If not, then start it
service cloudera-scm-agent start

Thanks,
Darren

Hi Darren,

Thank you for your quick reply.

# service cloudera-scm-agent status
cloudera-scm-agent (pid 3433) is running...

Rex Zhen

To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.

Hi Rex,

Do the agent logs (in /var/log/cloudera-scm-agent/ ) indicate any error
connecting to CM server? Make sure you're checking while ssh'd into the
host with the problem.

Thanks,
Darren

./cloudera-scm-agent.log:[12/Feb/2014 12:58:08 +0000] 12917 MainThread
agent ERROR Failed to contact supervisor after 635 attempts.
Agent will exit.
./cloudera-scm-agent.log:[12/Feb/2014 12:58:08 +0000] 12917 MainThread
agent ERROR Caught unexpected exception in main loop.
./cloudera-scm-agent.log:[12/Feb/2014 12:58:22 +0000] 12917
Monitor-RegionServerMonitor abstract_monitor ERROR Error fetching
metrics at 'http://dn-03.nim.com:60030/metrics?format=json'
./cloudera-scm-agent.log: raise URLError(err)
./cloudera-scm-agent.log:URLError: <urlopen error [Errno 111] Connection
refused>
./cloudera-scm-agent.log:[12/Feb/2014 12:58:23 +0000] 12917 MainThread
agent ERROR Could not contact supervisor.
./cloudera-scm-agent.log: raise error, msg
./cloudera-scm-agent.log:error: [Errno 111] Connection refused
./cloudera-scm-agent.log:[12/Feb/2014 12:58:23 +0000] 12917 MainThread
agent ERROR Failed to contact supervisor after 636 attempts.
Agent will exit.
./cloudera-scm-agent.log:[12/Feb/2014 12:58:23 +0000] 12917 MainThread
agent ERROR Caught unexpected exception in main loop.
./cloudera-scm-agent.log:[12/Feb/2014 12:58:32 +0000] 12917
MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to read file
descriptor max for process 14391: [Errno 2] No such file or directory:
'/proc/14391/limits'
./cloudera-scm-agent.log:[12/Feb/2014 12:58:32 +0000] 12917
MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to get file
descriptor count for process 14391: [Errno 2] No such file or directory:
'/proc/14391/fd/'
./cloudera-scm-agent.log:[12/Feb/2014 12:58:32 +0000] 12917
MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to get process
metrics 14391: no process found with pid 14391
./cloudera-scm-agent.log:[12/Feb/2014 12:58:38 +0000] 12917 MainThread
agent ERROR Could not contact supervisor.
./cloudera-scm-agent.log: raise error, msg
./cloudera-scm-agent.log:error: [Errno 111] Connection refused
./cloudera-scm-agent.log:[12/Feb/2014 12:58:38 +0000] 12917 MainThread
agent ERROR Failed to contact supervisor after 637 attempts.
Agent will exit.
./cloudera-scm-agent.log:[12/Feb/2014 12:58:38 +0000] 12917 MainThread
agent ERROR Caught unexpected exception in main loop.
./cloudera-scm-agent.log:[12/Feb/2014 12:58:53 +0000] 12917 MainThread
agent ERROR Could not contact supervisor.
./cloudera-scm-agent.log: raise error, msg
./cloudera-scm-agent.log:error: [Errno 111] Connection refused
./cloudera-scm-agent.log:[12/Feb/2014 12:58:53 +0000] 12917 MainThread
agent ERROR Failed to contact supervisor after 638 attempts.
Agent will exit.
./cloudera-scm-agent.log:[12/Feb/2014 12:58:53 +0000] 12917 MainThread
agent ERROR Caught unexpected exception in main loop.
./cloudera-scm-agent.log:[12/Feb/2014 12:59:08 +0000] 12917 MainThread
agent ERROR Could not contact supervisor.
./cloudera-scm-agent.log: raise error, msg
./cloudera-scm-agent.log:error: [Errno 111] Connection refused
./cloudera-scm-agent.log:[12/Feb/2014 12:59:08 +0000] 12917 MainThread
agent ERROR Failed to contact supervisor after 639 attempts.
Agent will exit.
./cloudera-scm-agent.log:[12/Feb/2014 12:59:08 +0000] 12917 MainThread
agent ERROR Caught unexpected exception in main loop.
./cloudera-scm-agent.log:[12/Feb/2014 14:09:13 +0000] 3433 MainThread
parcel ERROR Failed to deactivate system symlinks for parcel
CDH-4.2.0-1.cdh4.2.0.p0.10: 1
./cloudera-scm-agent.log:[12/Feb/2014 14:09:13 +0000] 3433 MainThread
parcel ERROR Failed to deactivate system symlinks for parcel
CDH-4.2.1-1.cdh4.2.1.p0.5: 1
./cloudera-scm-agent.log:[12/Feb/2014 14:45:54 +0000] 3433 MainThread
abstract_monitor ERROR JournalNodeMonitor for None unable to find
appropriate process to monitor for pid 11552.
./cloudera-scm-agent.out:Error: could not find config file
/var/run/cloudera-scm-agent/supervisor/supervisord.conf
./cmf_listener.log:ERROR:__main__:Error in supervisord listener loop.
./cmf_listener.log:OSError: [Errno 32] Broken pipe
./cmf_listener.log:ERROR:__main__:Error in supervisord listener loop.
./cmf_listener.log:OSError: [Errno 32] Broken pipe
./cmf_listener.log:ERROR:__main__:Error in supervisord listener loop.
./cmf_listener.log:OSError: [Errno 32] Broken pipe
./cmf_listener.log:ERROR:__main__:Error in supervisord listener loop.
./cmf_listener.log:OSError: [Errno 32] Broken pipe
./cmf_listener.log:ERROR:__main__:Error in supervisord listener loop.
./cmf_listener.log:OSError: [Errno 32] Broken pipe
./cmf_listener.log:ERROR:__main__:Error in supervisord listener loop.
./cmf_listener.log:OSError: [Errno 32] Broken pipe
To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.

Rex,

Are you running CM 4.6.x? There's a bug (fixed in CM 4.7) where the agent
doesn't properly exit if it can't contact supervisor. It looks like you've
been hit by that bug. I think the net result is that you have two agents
running.

To fix this, kill all the agents and all the supervisors on the machine.
Kill the agents with "kill -9", as that'll avoid provoking the above bug.
You should try killing the supervisors with a regular SIGTERM (not "-9") so
that any managed processes are properly killed as well.

Once you're done, use "service cloudera-scm-agent start" to restart the
agent.

To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.

When I checked the Parcels, the actived CDH is CDH 4.3.0-1.cdh4.3.0.p0.22 ,
does it have the same bug as 4.6.x?

To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.

That's the CDH version, not the CM version.

The easiest way to check the CM version is to look at the version of the
cloudera-manager-agent package (via rpm or dpkg, depending on your distro).

To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.

Yeah, you are right. the CM version is 4.6.3

cleaned up the process and restart the agent. most of the services are up
except the tasktracker.

here is the error log in the tasktracker,

2014-02-12 16:54:44,734 ERROR org.apache.hadoop.mapred.JettyBugMonitor:
Jetty bug monitor failed
2014-02-12 16:59:05,418 ERROR org.apache.hadoop.mapred.TaskTracker:
RECEIVED SIGNAL 15: SIGTERM

the tasktracker keep restarting itself.

To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.

Hmm, never seen that one before. Maybe try e-mailing the cdh-user list?

To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.

Jetty bug monitor triggered my memory. See
https://github.com/cloudera/hadoop-common/blob/cdh5-2.2.0_5.0.0b2/hadoop-mapreduce1-project/src/mapred/org/apache/hadoop/mapred/JettyBugMonitor.java.
  It was introduced in
https://issues.apache.org/jira/browse/MAPREDUCE-3184. In short, a bug in
how the JVM and Jetty interact causes things to go haywire, and the TT
kills itself. One TT dead is better than one TT slow, because the slow TT
would slow down everyone disproportionally. CM can be configured to
restart the TT after it dies.

It seems a bit unusual that this happens in a loop for your TT. I'd be
interested in your JVM and CDH combination, but at that point, as Adar
suggests, the cdh-user@ is a better bet. There are folks on there who know
many more details about this.

-- Philip



To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.