FAQ
All,

Just recently we started seeing an error crop back up in workflows that are
spawned by our coordinator jobs and I can't seem to pinpoint the issue. For
a while this was cleared up by adding some additional parameters to
our oozie.service.StoreService.jdbc.url as noted in
https://ccp.cloudera.com/display/CDHDOC/Known+Issues+and+Work+Arounds+in+CDH3#KnownIssuesandWorkAroundsinCDH3-Oozieki
and
the metrics data from JMX isn't showing anything out of the ordinary during
these spurts.

Please let me know if there is anything else that would be helpful to
include.

--
Matt

---- oozie-site.xml ----
<property>
<name>oozie.service.StoreService.jdbc.url</name>

<value>jdbc:mysql://<hostname>:3306/oozie?createDatabaseIfNotExist=true,TestOnBorrow=true,TestOnReturn=false,TestWhileIdle=false,ValidationQuery=select
1</value>
</property>

---- stacktrace ----
2012-09-28 11:58:31,367 WARN JavaActionExecutor:528 - USER[etl]
GROUP[users] TOKEN[] APP[ETL-ExtractionBranch]
JOB[0000901-120924114950063-oozie-oozi-W]
ACTION[0000901-120924114950063-oozie-oozi-W@initialize-node8] Launcher
ERROR, reason: Main class
[com.riotgames.dataservices.ETL.oozie.InitializeAction], main() threw
exception, Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The
driver has not received any packets from the server.
2012-09-28 11:58:31,367 WARN JavaActionExecutor:528 - USER[etl]
GROUP[users] TOKEN[] APP[ETL-ExtractionBranch]
JOB[0000901-120924114950063-oozie-oozi-W]
ACTION[0000901-120924114950063-oozie-oozi-W@initialize-node8] Launcher
exception: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The
driver has not received any packets from the server.
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
link failure

The last packet sent successfully to the server was 0 milliseconds ago. The
driver has not received any packets from the server.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at
com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1117)
at com.mysql.jdbc.MysqlIO.(ConnectionImpl.java:2393)
at
com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2430)
at
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2215)
at com.mysql.jdbc.ConnectionImpl.(JDBC4Connection.java:47)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:399)
at
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:334)
at java.sql.DriverManager.getConnection(DriverManager.java:582)
at java.sql.DriverManager.getConnection(DriverManager.java:185)
at
com.riotgames.dataservices.ETL.oozie.ConnectionFactory.createConnection(ConnectionFactory.java:26)
at
com.riotgames.dataservices.ETL.oozie.InitializeAction.main(InitializeAction.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:391)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at
java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at java.net.Socket.connect(Socket.java:478)
at java.net.Socket.(Socket.java:218)
at
com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:257)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:300)
... 29 more

--

Search Discussions

  • Raj Vishwanathan at Sep 28, 2012 at 7:37 pm
    Matt


    You probably have a network configuration or firewall  problem.

    Here is the relevant error message.

    "Caused by: java.net.ConnectException: Connection refused "


    Raj




    ________________________________
    From: Matt Goeke <goeke.matthew@gmail.com>
    To: cdh-user@cloudera.org
    Sent: Friday, September 28, 2012 12:15 PM
    Subject: Oozie cdh3u3: Communcations link failure


    All,


    Just recently we started seeing an error crop back up in workflows that are spawned by our coordinator jobs and I can't seem to pinpoint the issue. For a while this was cleared up by adding some additional parameters to our oozie.service.StoreService.jdbc.url as noted in https://ccp.cloudera.com/display/CDHDOC/Known+Issues+and+Work+Arounds+in+CDH3#KnownIssuesandWorkAroundsinCDH3-Oozieki and the metrics data from JMX isn't showing anything out of the ordinary during these spurts.


    Please let me know if there is anything else that would be helpful to include.


    --
    Matt


    ---- oozie-site.xml ----
    <property>
    <name>oozie.service.StoreService.jdbc.url</name>
    <value>jdbc:mysql://<hostname>:3306/oozie?createDatabaseIfNotExist=true,TestOnBorrow=true,TestOnReturn=false,TestWhileIdle=false,ValidationQuery=select 1</value>
    </property>


    ---- stacktrace ----
    2012-09-28 11:58:31,367  WARN JavaActionExecutor:528 - USER[etl] GROUP[users] TOKEN[] APP[ETL-ExtractionBranch] JOB[0000901-120924114950063-oozie-oozi-W] ACTION[0000901-120924114950063-oozie-oozi-W@initialize-node8] Launcher ERROR, reason: Main class [com.riotgames.dataservices.ETL.oozie.InitializeAction], main() threw exception, Communications link failure


    The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
    2012-09-28 11:58:31,367  WARN JavaActionExecutor:528 - USER[etl] GROUP[users] TOKEN[] APP[ETL-ExtractionBranch] JOB[0000901-120924114950063-oozie-oozi-W] ACTION[0000901-120924114950063-oozie-oozi-W@initialize-node8] Launcher exception: Communications link failure


    The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
    com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure


    The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
    at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1117)
    at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:350)
    at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2393)
    at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2430)
    at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2215)
    at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:813)
    at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
    at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:399)
    at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:334)
    at java.sql.DriverManager.getConnection(DriverManager.java:582)
    at java.sql.DriverManager.getConnection(DriverManager.java:185)
    at com.riotgames.dataservices.ETL.oozie.ConnectionFactory.createConnection(ConnectionFactory.java:26)
    at com.riotgames.dataservices.ETL.oozie.InitializeAction.main(InitializeAction.java:34)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:391)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
    at org.apache.hadoop.mapred.Child.main(Child.java:264)
    Caused by: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
    at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
    at java.net.Socket.connect(Socket.java:529)
    at java.net.Socket.connect(Socket.java:478)
    at java.net.Socket.<init>(Socket.java:375)
    at java.net.Socket.<init>(Socket.java:218)
    at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:257)
    at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:300)
    ... 29 more --



    --
  • Matt Goeke at Sep 28, 2012 at 8:54 pm
    Raj,

    As much as I would like for that to be the case I am doubting it is that
    straightforward in this instance. The Oozie metastore DB is actually local
    to the same box the Oozie server is currently running on and this is
    intermittent (i.e. I can submit 2 coordinator reruns at the same time and 1
    will fail on every node with the link error whereas the other will work
    perfectly). My best guess is this has to do with stale connections but the
    additional parameters to appended to the JDBC url should have resolved
    this. The only change I made to it was to take:

    ${ORIGINAL_JDBC_URL},TestOnBorrow=true,TestOnReturn=false,TestWhileIdle=false, ValidationQuery=select
    count(*) from SLA_EVENTS
    and change it to

    ${ORIGINAL_JDBC_URL},TestOnBorrow=true,TestOnReturn=false,TestWhileIdle=false, ValidationQuery=select
    1
    in order to make our provisioning with Chef easier but that should not
    affect the efficacy of the ValidationQuery as it will still need to resolve
    the DB itself. Also the jdbc connection does use a host name but I have
    verified that nslookup is correctly resolving using DNS.

    --
    Matt
    On Friday, September 28, 2012 2:31:34 PM UTC-5, Raj wrote:

    Matt


    You probably have a network configuration or firewall problem.

    Here is the relevant error message.

    "Caused by: java.net.ConnectException: Connection refused "


    Raj
    --

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcdh-user @
categorieshadoop
postedSep 28, '12 at 7:21p
activeSep 28, '12 at 8:54p
posts3
users3
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2018 Grokbase