FAQ
December 2 2012 4:15 PM Time Log Level Source Log Message 4:09:34.878 PM
INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

Reading configuration from:
/run/cloudera-scm-agent/process/168-zookeeper-server/zoo.cfg

  4:09:34.886 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

Invalid configuration, only one server specified (ignoring)

  4:09:34.899 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

autopurge.snapRetainCount set to 5

  4:09:34.900 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

autopurge.purgeInterval set to 24

  4:09:34.902 PM WARN org.apache.zookeeper.server.quorum.QuorumPeerMain

Either no config or no quorum defined in config, running in standalone mode

  4:09:34.909 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

Purge task started.

  4:09:34.953 PM ERROR org.apache.zookeeper.server.DatadirCleanupManager

Error occured while purging.
org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException:
Missing data directory /var/lib/zookeeper/version-2, automatic data
directory creation is disabled (zookeeper.datadir.autocreate is
false). Please create this directory manually.
  at org.apache.zookeeper.server.persistence.FileTxnSnapLog.(PurgeTxnLog.java:68)
  at org.apache.zookeeper.server.DatadirCleanupManager$PurgeTask.run(DatadirCleanupManager.java:140)
  at java.util.TimerThread.mainLoop(Timer.java:512)
  at java.util.TimerThread.run(Timer.java:462)

  4:09:34.956 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

Purge task completed.

  4:09:34.990 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

Reading configuration from:
/run/cloudera-scm-agent/process/168-zookeeper-server/zoo.cfg

  4:09:34.991 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

Invalid configuration, only one server specified (ignoring)

  4:09:34.992 PM INFO org.apache.zookeeper.server.ZooKeeperServerMain

Starting server

  4:09:35.020 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:zookeeper.version=3.4.3-cdh4.1.2--1, built on
11/01/2012 23:59 GMT

  4:09:35.020 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:host.name=slave4.medcloud.com

  4:09:35.020 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.version=1.6.0_31

  4:09:35.021 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.vendor=Sun Microsystems Inc.

  4:09:35.021 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.home=/usr/lib/jvm/j2sdk1.6-oracle/jre

  4:09:35.021 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.class.path=/run/cloudera-scm-agent/process/168-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.4.3-cdh4.1.2.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar:/usr/lib/zookeeper/lib/netty-3.2.2.Final.jar:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar

  4:09:35.021 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.library.path=/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64/server:/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64:/usr/lib/jvm/j2sdk1.6-oracle/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

  4:09:35.021 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.io.tmpdir=/tmp

  4:09:35.021 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.compiler=<NA>

  4:09:35.021 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:os.name=Linux

  4:09:35.022 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:os.arch=amd64

  4:09:35.022 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:os.version=3.2.0-29-generic

  4:09:35.022 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:user.name=zookeeper

  4:09:35.022 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:user.home=/var/lib/zookeeper

  4:09:35.022 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:user.dir=/run/cloudera-scm-agent/process/168-zookeeper-server

  4:09:35.025 PM ERROR org.apache.zookeeper.server.ZooKeeperServerMain

Unable to access datadir, exiting abnormally
org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException:
Missing data directory /var/lib/zookeeper/version-2, automatic data
directory creation is disabled (zookeeper.datadir.autocreate is
false). Please create this directory manually.
  at org.apache.zookeeper.server.persistence.FileTxnSnapLog.(ZooKeeperServerMain.java:108)
  at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:91)
  at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:53)
  at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:121)
  at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)

  4:09:36.939 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

Reading configuration from:
/run/cloudera-scm-agent/process/168-zookeeper-server/zoo.cfg

  4:09:36.945 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

Invalid configuration, only one server specified (ignoring)

  4:09:36.948 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

autopurge.snapRetainCount set to 5

  4:09:36.948 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

autopurge.purgeInterval set to 24

  4:09:36.950 PM WARN org.apache.zookeeper.server.quorum.QuorumPeerMain

Either no config or no quorum defined in config, running in standalone mode

  4:09:36.963 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

Purge task started.

  4:09:36.987 PM ERROR org.apache.zookeeper.server.DatadirCleanupManager

Error occured while purging.
org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException:
Missing data directory /var/lib/zookeeper/version-2, automatic data
directory creation is disabled (zookeeper.datadir.autocreate is
false). Please create this directory manually.
  at org.apache.zookeeper.server.persistence.FileTxnSnapLog.(PurgeTxnLog.java:68)
  at org.apache.zookeeper.server.DatadirCleanupManager$PurgeTask.run(DatadirCleanupManager.java:140)
  at java.util.TimerThread.mainLoop(Timer.java:512)
  at java.util.TimerThread.run(Timer.java:462)

  4:09:36.990 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

Purge task completed.

  4:09:36.997 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

Reading configuration from:
/run/cloudera-scm-agent/process/168-zookeeper-server/zoo.cfg

  4:09:36.999 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

Invalid configuration, only one server specified (ignoring)

  4:09:36.999 PM INFO org.apache.zookeeper.server.ZooKeeperServerMain

Starting server

  4:09:37.029 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:zookeeper.version=3.4.3-cdh4.1.2--1, built on
11/01/2012 23:59 GMT

  4:09:37.030 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:host.name=slave4.medcloud.com

  4:09:37.030 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.version=1.6.0_31

  4:09:37.030 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.vendor=Sun Microsystems Inc.

  4:09:37.030 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.home=/usr/lib/jvm/j2sdk1.6-oracle/jre

  4:09:37.030 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.class.path=/run/cloudera-scm-agent/process/168-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.4.3-cdh4.1.2.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar:/usr/lib/zookeeper/lib/netty-3.2.2.Final.jar:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar

  4:09:37.030 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.library.path=/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64/server:/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64:/usr/lib/jvm/j2sdk1.6-oracle/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

  4:09:37.030 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.io.tmpdir=/tmp

  4:09:37.031 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.compiler=<NA>

  4:09:37.031 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:os.name=Linux

  4:09:37.031 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:os.arch=amd64

  4:09:37.031 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:os.version=3.2.0-29-generic

  4:09:37.031 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:user.name=zookeeper

  4:09:37.031 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:user.home=/var/lib/zookeeper

  4:09:37.031 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:user.dir=/run/cloudera-scm-agent/process/168-zookeeper-server

  4:09:37.033 PM ERROR org.apache.zookeeper.server.ZooKeeperServerMain

Unable to access datadir, exiting abnormally
org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException:
Missing data directory /var/lib/zookeeper/version-2, automatic data
directory creation is disabled (zookeeper.datadir.autocreate is
false). Please create this directory manually.
  at org.apache.zookeeper.server.persistence.FileTxnSnapLog.(ZooKeeperServerMain.java:108)
  at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:91)
  at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:53)
  at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:121)
  at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)

  4:09:39.957 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

Reading configuration from:
/run/cloudera-scm-agent/process/168-zookeeper-server/zoo.cfg

  4:09:39.963 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

Invalid configuration, only one server specified (ignoring)

  4:09:39.966 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

autopurge.snapRetainCount set to 5

  4:09:39.966 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

autopurge.purgeInterval set to 24

  4:09:39.968 PM WARN org.apache.zookeeper.server.quorum.QuorumPeerMain

Either no config or no quorum defined in config, running in standalone mode

  4:09:39.975 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

Purge task started.

  4:09:40.005 PM ERROR org.apache.zookeeper.server.DatadirCleanupManager

Error occured while purging.
org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException:
Missing data directory /var/lib/zookeeper/version-2, automatic data
directory creation is disabled (zookeeper.datadir.autocreate is
false). Please create this directory manually.
  at org.apache.zookeeper.server.persistence.FileTxnSnapLog.(PurgeTxnLog.java:68)
  at org.apache.zookeeper.server.DatadirCleanupManager$PurgeTask.run(DatadirCleanupManager.java:140)
  at java.util.TimerThread.mainLoop(Timer.java:512)
  at java.util.TimerThread.run(Timer.java:462)

  4:09:40.007 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

Purge task completed.

  4:09:40.027 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

Reading configuration from:
/run/cloudera-scm-agent/process/168-zookeeper-server/zoo.cfg

  4:09:40.028 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

Invalid configuration, only one server specified (ignoring)

  4:09:40.028 PM INFO org.apache.zookeeper.server.ZooKeeperServerMain

Starting server

  4:09:40.050 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:zookeeper.version=3.4.3-cdh4.1.2--1, built on
11/01/2012 23:59 GMT

  4:09:40.051 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:host.name=slave4.medcloud.com

  4:09:40.051 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.version=1.6.0_31

  4:09:40.051 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.vendor=Sun Microsystems Inc.

  4:09:40.051 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.home=/usr/lib/jvm/j2sdk1.6-oracle/jre

  4:09:40.051 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.class.path=/run/cloudera-scm-agent/process/168-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.4.3-cdh4.1.2.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar:/usr/lib/zookeeper/lib/netty-3.2.2.Final.jar:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar

  4:09:40.051 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.library.path=/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64/server:/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64:/usr/lib/jvm/j2sdk1.6-oracle/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

  4:09:40.051 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.io.tmpdir=/tmp

  4:09:40.051 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.compiler=<NA>

  4:09:40.052 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:os.name=Linux

  4:09:40.052 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:os.arch=amd64

  4:09:40.052 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:os.version=3.2.0-29-generic

  4:09:40.052 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:user.name=zookeeper

  4:09:40.052 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:user.home=/var/lib/zookeeper

  4:09:40.052 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:user.dir=/run/cloudera-scm-agent/process/168-zookeeper-server

  4:09:40.054 PM ERROR org.apache.zookeeper.server.ZooKeeperServerMain

Unable to access datadir, exiting abnormally
org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException:
Missing data directory /var/lib/zookeeper/version-2, automatic data
directory creation is disabled (zookeeper.datadir.autocreate is
false). Please create this directory manually.
  at org.apache.zookeeper.server.persistence.FileTxnSnapLog.(ZooKeeperServerMain.java:108)
  at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:91)
  at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:53)
  at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:121)
  at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)

  4:09:43.969 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

Reading configuration from:
/run/cloudera-scm-agent/process/168-zookeeper-server/zoo.cfg

  4:09:43.975 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

Invalid configuration, only one server specified (ignoring)

  4:09:43.978 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

autopurge.snapRetainCount set to 5

  4:09:43.978 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

autopurge.purgeInterval set to 24

  4:09:43.980 PM WARN org.apache.zookeeper.server.quorum.QuorumPeerMain

Either no config or no quorum defined in config, running in standalone mode

  4:09:43.987 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

Purge task started.

  4:09:44.015 PM ERROR org.apache.zookeeper.server.DatadirCleanupManager

Error occured while purging.
org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException:
Missing data directory /var/lib/zookeeper/version-2, automatic data
directory creation is disabled (zookeeper.datadir.autocreate is
false). Please create this directory manually.
  at org.apache.zookeeper.server.persistence.FileTxnSnapLog.(PurgeTxnLog.java:68)
  at org.apache.zookeeper.server.DatadirCleanupManager$PurgeTask.run(DatadirCleanupManager.java:140)
  at java.util.TimerThread.mainLoop(Timer.java:512)
  at java.util.TimerThread.run(Timer.java:462)

  4:09:44.017 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

Purge task completed.

  4:09:44.039 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

Reading configuration from:
/run/cloudera-scm-agent/process/168-zookeeper-server/zoo.cfg

  4:09:44.041 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

Invalid configuration, only one server specified (ignoring)

  4:09:44.041 PM INFO org.apache.zookeeper.server.ZooKeeperServerMain

Starting server

  4:09:44.063 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:zookeeper.version=3.4.3-cdh4.1.2--1, built on
11/01/2012 23:59 GMT

  4:09:44.063 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:host.name=slave4.medcloud.com

  4:09:44.064 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.version=1.6.0_31

  4:09:44.064 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.vendor=Sun Microsystems Inc.

  4:09:44.064 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.home=/usr/lib/jvm/j2sdk1.6-oracle/jre

  4:09:44.064 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.class.path=/run/cloudera-scm-agent/process/168-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.4.3-cdh4.1.2.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar:/usr/lib/zookeeper/lib/netty-3.2.2.Final.jar:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar

  4:09:44.064 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.library.path=/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64/server:/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64:/usr/lib/jvm/j2sdk1.6-oracle/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

  4:09:44.064 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.io.tmpdir=/tmp

  4:09:44.064 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.compiler=<NA>

  4:09:44.064 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:os.name=Linux

  4:09:44.064 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:os.arch=amd64

  4:09:44.065 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:os.version=3.2.0-29-generic

  4:09:44.065 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:user.name=zookeeper

  4:09:44.065 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:user.home=/var/lib/zookeeper

  4:09:44.065 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:user.dir=/run/cloudera-scm-agent/process/168-zookeeper-server

  4:09:44.067 PM ERROR org.apache.zookeeper.server.ZooKeeperServerMain

Unable to access datadir, exiting abnormally
org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException:
Missing data directory /var/lib/zookeeper/version-2, automatic data
directory creation is disabled (zookeeper.datadir.autocreate is
false). Please create this directory manually.
  at org.apache.zookeeper.server.persistence.FileTxnSnapLog.(ZooKeeperServerMain.java:108)
  at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:91)
  at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:53)
  at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:121)
  at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)

  4:10:18.647 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

Reading configuration from:
/run/cloudera-scm-agent/process/170-zookeeper-server/zoo.cfg

  4:10:18.653 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

Invalid configuration, only one server specified (ignoring)

  4:10:18.656 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

autopurge.snapRetainCount set to 5

  4:10:18.656 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

autopurge.purgeInterval set to 24

  4:10:18.658 PM WARN org.apache.zookeeper.server.quorum.QuorumPeerMain

Either no config or no quorum defined in config, running in standalone mode

  4:10:18.667 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

Purge task started.

  4:10:18.717 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

Purge task completed.

  4:10:18.726 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

Reading configuration from:
/run/cloudera-scm-agent/process/170-zookeeper-server/zoo.cfg

  4:10:18.727 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

Invalid configuration, only one server specified (ignoring)

  4:10:18.727 PM INFO org.apache.zookeeper.server.ZooKeeperServerMain

Starting server

  4:10:18.748 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:zookeeper.version=3.4.3-cdh4.1.2--1, built on
11/01/2012 23:59 GMT

  4:10:18.748 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:host.name=slave4.medcloud.com

  4:10:18.748 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.version=1.6.0_31

  4:10:18.748 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.vendor=Sun Microsystems Inc.

  4:10:18.749 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.home=/usr/lib/jvm/j2sdk1.6-oracle/jre

  4:10:18.749 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.class.path=/run/cloudera-scm-agent/process/170-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.4.3-cdh4.1.2.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar:/usr/lib/zookeeper/lib/netty-3.2.2.Final.jar:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar

  4:10:18.749 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.library.path=/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64/server:/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64:/usr/lib/jvm/j2sdk1.6-oracle/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

  4:10:18.749 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.io.tmpdir=/tmp

  4:10:18.749 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:java.compiler=<NA>

  4:10:18.749 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:os.name=Linux

  4:10:18.749 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:os.arch=amd64

  4:10:18.749 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:os.version=3.2.0-29-generic

  4:10:18.749 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:user.name=zookeeper

  4:10:18.750 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:user.home=/var/lib/zookeeper

  4:10:18.750 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:user.dir=/run/cloudera-scm-agent/process/170-zookeeper-server

  4:10:18.752 PM INFO org.apache.zookeeper.server.ZooKeeperServer

tickTime set to 2000

  4:10:18.752 PM INFO org.apache.zookeeper.server.ZooKeeperServer

minSessionTimeout set to 4000

  4:10:18.752 PM INFO org.apache.zookeeper.server.ZooKeeperServer

maxSessionTimeout set to 40000

  4:10:18.838 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

binding to port 0.0.0.0/0.0.0.0:2181

  4:10:18.888 PM INFO org.apache.zookeeper.server.persistence.FileTxnSnapLog

Snapshotting: 0x0 to /var/lib/zookeeper/version-2/snapshot.0

  4:14:10.689 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

Accepted socket connection from /192.168.0.8:39761

  4:14:10.738 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Client attempting to establish new session at /192.168.0.8:39761

  4:14:10.764 PM INFO org.apache.zookeeper.server.persistence.FileTxnLog

Creating new log file: log.1

  4:14:10.868 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Established session 0x13b5bf40c940000 with negotiated timeout 40000
for client /192.168.0.8:39761

  4:14:11.028 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

Accepted socket connection from /192.168.0.5:35186

  4:14:11.041 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Client attempting to establish new session at /192.168.0.5:35186

  4:14:11.050 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Established session 0x13b5bf40c940001 with negotiated timeout 40000
for client /192.168.0.5:35186

  4:14:11.161 PM INFO org.apache.zookeeper.server.PrepRequestProcessor

Got user-level KeeperException when processing
sessionid:0x13b5bf40c940000 type:delete cxid:0x11 zxid:0xb txntype:-1
reqpath:n/a Error
Path:/hbase/backup-masters/slave7.medcloud.com,60000,1354457649683
Error:KeeperErrorCode = NoNode for
/hbase/backup-masters/slave7.medcloud.com,60000,1354457649683

  4:14:12.589 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

Accepted socket connection from /192.168.0.4:48929

  4:14:12.618 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Client attempting to establish new session at /192.168.0.4:48929

  4:14:12.633 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Established session 0x13b5bf40c940002 with negotiated timeout 40000
for client /192.168.0.4:48929

  4:14:13.654 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

Accepted socket connection from /192.168.0.2:59878

  4:14:13.667 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Client attempting to establish new session at /192.168.0.2:59878

  4:14:13.683 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Established session 0x13b5bf40c940003 with negotiated timeout 40000
for client /192.168.0.2:59878

  4:14:14.165 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

Accepted socket connection from /192.168.0.8:39764

  4:14:14.171 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Client attempting to establish new session at /192.168.0.8:39764

  4:14:14.175 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Established session 0x13b5bf40c940004 with negotiated timeout 40000
for client /192.168.0.8:39764

  4:14:14.297 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

Accepted socket connection from /192.168.0.5:35187

  4:14:14.299 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Client attempting to establish new session at /192.168.0.5:35187

  4:14:14.309 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Established session 0x13b5bf40c940005 with negotiated timeout 40000
for client /192.168.0.5:35187

  4:14:14.310 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

Accepted socket connection from /192.168.0.4:48930

  4:14:14.311 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Client attempting to establish new session at /192.168.0.4:48930

  4:14:14.316 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Established session 0x13b5bf40c940006 with negotiated timeout 40000
for client /192.168.0.4:48930

  4:14:14.543 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

Accepted socket connection from /192.168.0.2:59879

  4:14:14.544 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Client attempting to establish new session at /192.168.0.2:59879

  4:14:14.558 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Established session 0x13b5bf40c940007 with negotiated timeout 40000
for client /192.168.0.2:59879

  4:14:14.563 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

Accepted socket connection from /192.168.0.3:47913

  4:14:14.588 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Client attempting to establish new session at /192.168.0.3:47913

  4:14:14.591 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Established session 0x13b5bf40c940008 with negotiated timeout 40000
for client /192.168.0.3:47913

  4:14:14.806 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

Accepted socket connection from /192.168.0.3:47914

  4:14:14.807 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Client attempting to establish new session at /192.168.0.3:47914

  4:14:14.816 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Established session 0x13b5bf40c940009 with negotiated timeout 40000
for client /192.168.0.3:47914

  4:14:21.841 PM INFO org.apache.zookeeper.server.PrepRequestProcessor

Got user-level KeeperException when processing
sessionid:0x13b5bf40c940000 type:delete cxid:0x30 zxid:0x1b txntype:-1
reqpath:n/a Error Path:/hbase/root-region-server Error:KeeperErrorCode
= NoNode for /hbase/root-region-server

  4:14:41.679 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

Reading configuration from:
/run/cloudera-scm-agent/process/183-zookeeper-server/zoo.cfg

  4:14:41.685 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

Invalid configuration, only one server specified (ignoring)

  4:14:41.688 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

autopurge.snapRetainCount set to 5

  4:14:41.688 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

autopurge.purgeInterval set to 24

  4:14:41.690 PM WARN org.apache.zookeeper.server.quorum.QuorumPeerMain

Either no config or no quorum defined in config, running in standalone mode

  4:14:41.699 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

Purge task started.


On Sun, Dec 2, 2012 at 1:41 PM, Shouvanik Haldar wrote:

In addition to my previous mail,
below are the log messages, when job tracker does not start

2012-12-02 06:36:26,190 INFO
[CommandPusher:service.AbstractBringUpBringDownCommands@505] BringUp
command (155) has finished unsuccessfully on service DbService{id=10,
name=mapreduce1, serviceType=MAPREDUCE, optimisticLockVersion=8} role id:28
name: mapreduce1-JOBTRACKER-2 hostId:ip-10-40-222-77.ec2.internal
roleType:JOBTRACKER configuredStatus:STOPPED configGeneration:6 status:NA
service:mapreduce1.


2012-12-02 00:02:53,680 WARN org.apache.hadoop.mapred.JobTracker: Error
starting tracker: ENOENT: No such file or directory
at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:521)
at
org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
at
org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
at org.apache.hadoop.mapred.JobHistory.init(JobHistory.java:335)
at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2178)
at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2175)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2175)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2050)
at
org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:296)
at
org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:288)
at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4792)



2012-12-02 00:02:54,739 FATAL org.apache.hadoop.mapred.JobTracker:
java.net.BindException: Problem binding to ip-10-40-222-77.ec2.internal/
10.40.222.77:8021 : Address already in use
at org.apache.hadoop.ipc.Server.bind(Server.java:230)
at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:319)
at org.apache.hadoop.ipc.Server.<init>(Server.java:1529)
at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:539)
at org.apache.hadoop.ipc.RPC.getServer(RPC.java:500)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2143)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2050)
at
org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:296)
at
org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:288)
at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4792)
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at org.apache.hadoop.ipc.Server.bind(Server.java:228)
... 9 more

2012-12-02 00:02:54,757 INFO org.apache.hadoop.mapred.JobTracker:
SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down JobTracker at ip-10-40-222-77.ec2.internal/
10.40.222.77


Please help.

Regards
Shouvanik


On Sun, Dec 2, 2012 at 4:20 PM, bc Wong wrote:
On Sun, Dec 2, 2012 at 12:05 AM, Dalia Hassan wrote:

I have been in this mess for a couple of days, when I try to solve a
problem another occurs, don't know why.

now when I start the zookeeper i get the following error although all
hosts are up.

Service did not start successfully; not all of the required roles
started: No ZooKeeper servers are running.
Dalia,

I noticed your frustration from your choice of subject title. And I
looked through the questions that you've posted in this mailing list. Most
threads have recommendations but not the final resolution. But from what I
can tell, you're having a fair amount of environmental issues (like DNS),
especially if you're running in a cloud. While Cloudera Manager tries to
eliminate the need to have special Hadoop knowledge and sysadmin expertise,
we're not there yet. Running a distributed system is fundamentally hard.
Everyone on the CM team works day and night to build solid user-friendly
features. The software can cover more error cases, and can give most useful
error messages, and we will make that happen.

For this specific problem, it seems that your ZK servers are not starting
right. Could you attach the ZK server log (go to CM -> ZK -> Server ->
Processes -> Role log)? And make sure that the agents are heartbeating (go
to CM -> Hosts).

Thanks for your patience.

Cheers,
bc


--
Thanks,
*Shouvanik*

Search Discussions

  • bc Wong at Dec 3, 2012 at 5:46 am

    On Sun, Dec 2, 2012 at 6:49 AM, Dalia Hassan wrote:

    December 2 2012 4:15 PM Time Log Level Source Log Message 4:09:34.878
    PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Reading configuration from: /run/cloudera-scm-agent/process/168-zookeeper-server/zoo.cfg

    4:09:34.886 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Invalid configuration, only one server specified (ignoring)

    4:09:34.899 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    autopurge.snapRetainCount set to 5

    4:09:34.900 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    autopurge.purgeInterval set to 24

    4:09:34.902 PM WARN org.apache.zookeeper.server.quorum.QuorumPeerMain

    Either no config or no quorum defined in config, running in standalone mode

    4:09:34.909 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    Purge task started.

    4:09:34.953 PM ERROR org.apache.zookeeper.server.DatadirCleanupManager

    Error occured while purging.
    org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Missing data directory /var/lib/zookeeper/version-2, automatic data directory creation is disabled (zookeeper.datadir.autocreate is false). Please create this directory manually.
    at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:97)
    at org.apache.zookeeper.server.PurgeTxnLog.purge(PurgeTxnLog.java:68)
    at org.apache.zookeeper.server.DatadirCleanupManager$PurgeTask.run(DatadirCleanupManager.java:140)
    at java.util.TimerThread.mainLoop(Timer.java:512)
    at java.util.TimerThread.run(Timer.java:462)
    Dalia,

    This is a separate problem from the one Shouvanik posted (which was a port
    conflict on 8021 for the JT). Here, the "missing data directory" seems to
    be the issue.

    Go to CM. Make sure that the ZK service is stopped. (It's probably not
    running to begin with.) Then go to ZK -> Server -> Actions -> Initialize.
    If the command succeeds, then you can restart your ZK. Otherwise, can you
    attach the failure message for the "Initialize" command?

    Cheers,
    bc

    4:09:34.956 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    Purge task completed.

    4:09:34.990 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Reading configuration from: /run/cloudera-scm-agent/process/168-zookeeper-server/zoo.cfg

    4:09:34.991 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Invalid configuration, only one server specified (ignoring)

    4:09:34.992 PM INFO org.apache.zookeeper.server.ZooKeeperServerMain

    Starting server

    4:09:35.020 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:zookeeper.version=3.4.3-cdh4.1.2--1, built on 11/01/2012 23:59 GMT

    4:09:35.020 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:host.name=slave4.medcloud.com

    4:09:35.020 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.version=1.6.0_31

    4:09:35.021 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.vendor=Sun Microsystems Inc.

    4:09:35.021 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.home=/usr/lib/jvm/j2sdk1.6-oracle/jre

    4:09:35.021 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.class.path=/run/cloudera-scm-agent/process/168-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.4.3-cdh4.1.2.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar:/usr/lib/zookeeper/lib/netty-3.2.2.Final.jar:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar

    4:09:35.021 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.library.path=/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64/server:/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64:/usr/lib/jvm/j2sdk1.6-oracle/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

    4:09:35.021 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.io.tmpdir=/tmp

    4:09:35.021 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.compiler=<NA>

    4:09:35.021 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:os.name=Linux

    4:09:35.022 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:os.arch=amd64

    4:09:35.022 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:os.version=3.2.0-29-generic

    4:09:35.022 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:user.name=zookeeper

    4:09:35.022 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:user.home=/var/lib/zookeeper

    4:09:35.022 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:user.dir=/run/cloudera-scm-agent/process/168-zookeeper-server

    4:09:35.025 PM ERROR org.apache.zookeeper.server.ZooKeeperServerMain

    Unable to access datadir, exiting abnormally
    org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Missing data directory /var/lib/zookeeper/version-2, automatic data directory creation is disabled (zookeeper.datadir.autocreate is false). Please create this directory manually.
    at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:97)
    at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:108)
    at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:91)
    at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:53)
    at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:121)
    at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)

    4:09:36.939 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Reading configuration from: /run/cloudera-scm-agent/process/168-zookeeper-server/zoo.cfg

    4:09:36.945 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Invalid configuration, only one server specified (ignoring)

    4:09:36.948 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    autopurge.snapRetainCount set to 5

    4:09:36.948 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    autopurge.purgeInterval set to 24

    4:09:36.950 PM WARN org.apache.zookeeper.server.quorum.QuorumPeerMain

    Either no config or no quorum defined in config, running in standalone mode

    4:09:36.963 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    Purge task started.

    4:09:36.987 PM ERROR org.apache.zookeeper.server.DatadirCleanupManager

    Error occured while purging.
    org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Missing data directory /var/lib/zookeeper/version-2, automatic data directory creation is disabled (zookeeper.datadir.autocreate is false). Please create this directory manually.
    at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:97)
    at org.apache.zookeeper.server.PurgeTxnLog.purge(PurgeTxnLog.java:68)
    at org.apache.zookeeper.server.DatadirCleanupManager$PurgeTask.run(DatadirCleanupManager.java:140)
    at java.util.TimerThread.mainLoop(Timer.java:512)
    at java.util.TimerThread.run(Timer.java:462)

    4:09:36.990 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    Purge task completed.

    4:09:36.997 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Reading configuration from: /run/cloudera-scm-agent/process/168-zookeeper-server/zoo.cfg

    4:09:36.999 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Invalid configuration, only one server specified (ignoring)

    4:09:36.999 PM INFO org.apache.zookeeper.server.ZooKeeperServerMain

    Starting server

    4:09:37.029 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:zookeeper.version=3.4.3-cdh4.1.2--1, built on 11/01/2012 23:59 GMT

    4:09:37.030 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:host.name=slave4.medcloud.com

    4:09:37.030 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.version=1.6.0_31

    4:09:37.030 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.vendor=Sun Microsystems Inc.

    4:09:37.030 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.home=/usr/lib/jvm/j2sdk1.6-oracle/jre

    4:09:37.030 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.class.path=/run/cloudera-scm-agent/process/168-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.4.3-cdh4.1.2.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar:/usr/lib/zookeeper/lib/netty-3.2.2.Final.jar:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar

    4:09:37.030 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.library.path=/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64/server:/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64:/usr/lib/jvm/j2sdk1.6-oracle/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

    4:09:37.030 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.io.tmpdir=/tmp

    4:09:37.031 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.compiler=<NA>

    4:09:37.031 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:os.name=Linux

    4:09:37.031 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:os.arch=amd64

    4:09:37.031 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:os.version=3.2.0-29-generic

    4:09:37.031 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:user.name=zookeeper

    4:09:37.031 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:user.home=/var/lib/zookeeper

    4:09:37.031 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:user.dir=/run/cloudera-scm-agent/process/168-zookeeper-server

    4:09:37.033 PM ERROR org.apache.zookeeper.server.ZooKeeperServerMain

    Unable to access datadir, exiting abnormally
    org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Missing data directory /var/lib/zookeeper/version-2, automatic data directory creation is disabled (zookeeper.datadir.autocreate is false). Please create this directory manually.
    at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:97)
    at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:108)
    at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:91)
    at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:53)
    at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:121)
    at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)

    4:09:39.957 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Reading configuration from: /run/cloudera-scm-agent/process/168-zookeeper-server/zoo.cfg

    4:09:39.963 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Invalid configuration, only one server specified (ignoring)

    4:09:39.966 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    autopurge.snapRetainCount set to 5

    4:09:39.966 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    autopurge.purgeInterval set to 24

    4:09:39.968 PM WARN org.apache.zookeeper.server.quorum.QuorumPeerMain

    Either no config or no quorum defined in config, running in standalone mode

    4:09:39.975 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    Purge task started.

    4:09:40.005 PM ERROR org.apache.zookeeper.server.DatadirCleanupManager

    Error occured while purging.
    org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Missing data directory /var/lib/zookeeper/version-2, automatic data directory creation is disabled (zookeeper.datadir.autocreate is false). Please create this directory manually.
    at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:97)
    at org.apache.zookeeper.server.PurgeTxnLog.purge(PurgeTxnLog.java:68)
    at org.apache.zookeeper.server.DatadirCleanupManager$PurgeTask.run(DatadirCleanupManager.java:140)
    at java.util.TimerThread.mainLoop(Timer.java:512)
    at java.util.TimerThread.run(Timer.java:462)

    4:09:40.007 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    Purge task completed.

    4:09:40.027 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Reading configuration from: /run/cloudera-scm-agent/process/168-zookeeper-server/zoo.cfg

    4:09:40.028 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Invalid configuration, only one server specified (ignoring)

    4:09:40.028 PM INFO org.apache.zookeeper.server.ZooKeeperServerMain

    Starting server

    4:09:40.050 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:zookeeper.version=3.4.3-cdh4.1.2--1, built on 11/01/2012 23:59 GMT

    4:09:40.051 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:host.name=slave4.medcloud.com

    4:09:40.051 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.version=1.6.0_31

    4:09:40.051 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.vendor=Sun Microsystems Inc.

    4:09:40.051 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.home=/usr/lib/jvm/j2sdk1.6-oracle/jre

    4:09:40.051 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.class.path=/run/cloudera-scm-agent/process/168-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.4.3-cdh4.1.2.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar:/usr/lib/zookeeper/lib/netty-3.2.2.Final.jar:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar

    4:09:40.051 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.library.path=/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64/server:/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64:/usr/lib/jvm/j2sdk1.6-oracle/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

    4:09:40.051 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.io.tmpdir=/tmp

    4:09:40.051 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.compiler=<NA>

    4:09:40.052 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:os.name=Linux

    4:09:40.052 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:os.arch=amd64

    4:09:40.052 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:os.version=3.2.0-29-generic

    4:09:40.052 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:user.name=zookeeper

    4:09:40.052 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:user.home=/var/lib/zookeeper

    4:09:40.052 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:user.dir=/run/cloudera-scm-agent/process/168-zookeeper-server

    4:09:40.054 PM ERROR org.apache.zookeeper.server.ZooKeeperServerMain

    Unable to access datadir, exiting abnormally
    org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Missing data directory /var/lib/zookeeper/version-2, automatic data directory creation is disabled (zookeeper.datadir.autocreate is false). Please create this directory manually.
    at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:97)
    at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:108)
    at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:91)
    at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:53)
    at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:121)
    at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)

    4:09:43.969 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Reading configuration from: /run/cloudera-scm-agent/process/168-zookeeper-server/zoo.cfg

    4:09:43.975 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Invalid configuration, only one server specified (ignoring)

    4:09:43.978 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    autopurge.snapRetainCount set to 5

    4:09:43.978 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    autopurge.purgeInterval set to 24

    4:09:43.980 PM WARN org.apache.zookeeper.server.quorum.QuorumPeerMain

    Either no config or no quorum defined in config, running in standalone mode

    4:09:43.987 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    Purge task started.

    4:09:44.015 PM ERROR org.apache.zookeeper.server.DatadirCleanupManager

    Error occured while purging.
    org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Missing data directory /var/lib/zookeeper/version-2, automatic data directory creation is disabled (zookeeper.datadir.autocreate is false). Please create this directory manually.
    at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:97)
    at org.apache.zookeeper.server.PurgeTxnLog.purge(PurgeTxnLog.java:68)
    at org.apache.zookeeper.server.DatadirCleanupManager$PurgeTask.run(DatadirCleanupManager.java:140)
    at java.util.TimerThread.mainLoop(Timer.java:512)
    at java.util.TimerThread.run(Timer.java:462)

    4:09:44.017 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    Purge task completed.

    4:09:44.039 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Reading configuration from: /run/cloudera-scm-agent/process/168-zookeeper-server/zoo.cfg

    4:09:44.041 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Invalid configuration, only one server specified (ignoring)

    4:09:44.041 PM INFO org.apache.zookeeper.server.ZooKeeperServerMain

    Starting server

    4:09:44.063 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:zookeeper.version=3.4.3-cdh4.1.2--1, built on 11/01/2012 23:59 GMT

    4:09:44.063 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:host.name=slave4.medcloud.com

    4:09:44.064 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.version=1.6.0_31

    4:09:44.064 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.vendor=Sun Microsystems Inc.

    4:09:44.064 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.home=/usr/lib/jvm/j2sdk1.6-oracle/jre

    4:09:44.064 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.class.path=/run/cloudera-scm-agent/process/168-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.4.3-cdh4.1.2.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar:/usr/lib/zookeeper/lib/netty-3.2.2.Final.jar:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar

    4:09:44.064 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.library.path=/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64/server:/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64:/usr/lib/jvm/j2sdk1.6-oracle/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

    4:09:44.064 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.io.tmpdir=/tmp

    4:09:44.064 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.compiler=<NA>

    4:09:44.064 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:os.name=Linux

    4:09:44.064 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:os.arch=amd64

    4:09:44.065 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:os.version=3.2.0-29-generic

    4:09:44.065 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:user.name=zookeeper

    4:09:44.065 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:user.home=/var/lib/zookeeper

    4:09:44.065 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:user.dir=/run/cloudera-scm-agent/process/168-zookeeper-server

    4:09:44.067 PM ERROR org.apache.zookeeper.server.ZooKeeperServerMain

    Unable to access datadir, exiting abnormally
    org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Missing data directory /var/lib/zookeeper/version-2, automatic data directory creation is disabled (zookeeper.datadir.autocreate is false). Please create this directory manually.
    at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:97)
    at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:108)
    at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:91)
    at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:53)
    at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:121)
    at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)

    4:10:18.647 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Reading configuration from: /run/cloudera-scm-agent/process/170-zookeeper-server/zoo.cfg

    4:10:18.653 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Invalid configuration, only one server specified (ignoring)

    4:10:18.656 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    autopurge.snapRetainCount set to 5

    4:10:18.656 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    autopurge.purgeInterval set to 24

    4:10:18.658 PM WARN org.apache.zookeeper.server.quorum.QuorumPeerMain

    Either no config or no quorum defined in config, running in standalone mode

    4:10:18.667 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    Purge task started.

    4:10:18.717 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    Purge task completed.

    4:10:18.726 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Reading configuration from: /run/cloudera-scm-agent/process/170-zookeeper-server/zoo.cfg

    4:10:18.727 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Invalid configuration, only one server specified (ignoring)

    4:10:18.727 PM INFO org.apache.zookeeper.server.ZooKeeperServerMain

    Starting server

    4:10:18.748 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:zookeeper.version=3.4.3-cdh4.1.2--1, built on 11/01/2012 23:59 GMT

    4:10:18.748 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:host.name=slave4.medcloud.com

    4:10:18.748 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.version=1.6.0_31

    4:10:18.748 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.vendor=Sun Microsystems Inc.

    4:10:18.749 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.home=/usr/lib/jvm/j2sdk1.6-oracle/jre

    4:10:18.749 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.class.path=/run/cloudera-scm-agent/process/170-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.4.3-cdh4.1.2.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar:/usr/lib/zookeeper/lib/netty-3.2.2.Final.jar:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar

    4:10:18.749 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.library.path=/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64/server:/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64:/usr/lib/jvm/j2sdk1.6-oracle/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

    4:10:18.749 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.io.tmpdir=/tmp

    4:10:18.749 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.compiler=<NA>

    4:10:18.749 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:os.name=Linux

    4:10:18.749 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:os.arch=amd64

    4:10:18.749 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:os.version=3.2.0-29-generic

    4:10:18.749 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:user.name=zookeeper

    4:10:18.750 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:user.home=/var/lib/zookeeper

    4:10:18.750 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:user.dir=/run/cloudera-scm-agent/process/170-zookeeper-server

    4:10:18.752 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    tickTime set to 2000

    4:10:18.752 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    minSessionTimeout set to 4000

    4:10:18.752 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    maxSessionTimeout set to 40000

    4:10:18.838 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

    binding to port 0.0.0.0/0.0.0.0:2181

    4:10:18.888 PM INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog

    Snapshotting: 0x0 to /var/lib/zookeeper/version-2/snapshot.0

    4:14:10.689 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

    Accepted socket connection from /192.168.0.8:39761

    4:14:10.738 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Client attempting to establish new session at /192.168.0.8:39761

    4:14:10.764 PM INFO org.apache.zookeeper.server.persistence.FileTxnLog

    Creating new log file: log.1

    4:14:10.868 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Established session 0x13b5bf40c940000 with negotiated timeout 40000 for client /192.168.0.8:39761

    4:14:11.028 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

    Accepted socket connection from /192.168.0.5:35186

    4:14:11.041 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Client attempting to establish new session at /192.168.0.5:35186

    4:14:11.050 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Established session 0x13b5bf40c940001 with negotiated timeout 40000 for client /192.168.0.5:35186

    4:14:11.161 PM INFO org.apache.zookeeper.server.PrepRequestProcessor

    Got user-level KeeperException when processing sessionid:0x13b5bf40c940000 type:delete cxid:0x11 zxid:0xb txntype:-1 reqpath:n/a Error Path:/hbase/backup-masters/slave7.medcloud.com,60000,1354457649683 Error:KeeperErrorCode = NoNode for /hbase/backup-masters/slave7.medcloud.com,60000,1354457649683

    4:14:12.589 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

    Accepted socket connection from /192.168.0.4:48929

    4:14:12.618 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Client attempting to establish new session at /192.168.0.4:48929

    4:14:12.633 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Established session 0x13b5bf40c940002 with negotiated timeout 40000 for client /192.168.0.4:48929

    4:14:13.654 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

    Accepted socket connection from /192.168.0.2:59878

    4:14:13.667 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Client attempting to establish new session at /192.168.0.2:59878

    4:14:13.683 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Established session 0x13b5bf40c940003 with negotiated timeout 40000 for client /192.168.0.2:59878

    4:14:14.165 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

    Accepted socket connection from /192.168.0.8:39764

    4:14:14.171 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Client attempting to establish new session at /192.168.0.8:39764

    4:14:14.175 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Established session 0x13b5bf40c940004 with negotiated timeout 40000 for client /192.168.0.8:39764

    4:14:14.297 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

    Accepted socket connection from /192.168.0.5:35187

    4:14:14.299 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Client attempting to establish new session at /192.168.0.5:35187

    4:14:14.309 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Established session 0x13b5bf40c940005 with negotiated timeout 40000 for client /192.168.0.5:35187

    4:14:14.310 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

    Accepted socket connection from /192.168.0.4:48930

    4:14:14.311 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Client attempting to establish new session at /192.168.0.4:48930

    4:14:14.316 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Established session 0x13b5bf40c940006 with negotiated timeout 40000 for client /192.168.0.4:48930

    4:14:14.543 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

    Accepted socket connection from /192.168.0.2:59879

    4:14:14.544 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Client attempting to establish new session at /192.168.0.2:59879

    4:14:14.558 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Established session 0x13b5bf40c940007 with negotiated timeout 40000 for client /192.168.0.2:59879

    4:14:14.563 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

    Accepted socket connection from /192.168.0.3:47913

    4:14:14.588 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Client attempting to establish new session at /192.168.0.3:47913

    4:14:14.591 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Established session 0x13b5bf40c940008 with negotiated timeout 40000 for client /192.168.0.3:47913

    4:14:14.806 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

    Accepted socket connection from /192.168.0.3:47914

    4:14:14.807 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Client attempting to establish new session at /192.168.0.3:47914

    4:14:14.816 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Established session 0x13b5bf40c940009 with negotiated timeout 40000 for client /192.168.0.3:47914

    4:14:21.841 PM INFO org.apache.zookeeper.server.PrepRequestProcessor

    Got user-level KeeperException when processing sessionid:0x13b5bf40c940000 type:delete cxid:0x30 zxid:0x1b txntype:-1 reqpath:n/a Error Path:/hbase/root-region-server Error:KeeperErrorCode = NoNode for /hbase/root-region-server

    4:14:41.679 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Reading configuration from: /run/cloudera-scm-agent/process/183-zookeeper-server/zoo.cfg

    4:14:41.685 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Invalid configuration, only one server specified (ignoring)

    4:14:41.688 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    autopurge.snapRetainCount set to 5

    4:14:41.688 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    autopurge.purgeInterval set to 24

    4:14:41.690 PM WARN org.apache.zookeeper.server.quorum.QuorumPeerMain

    Either no config or no quorum defined in config, running in standalone mode

    4:14:41.699 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    Purge task started.



    On Sun, Dec 2, 2012 at 1:41 PM, Shouvanik Haldar <
    shouvanik.haldar@gmail.com> wrote:
    In addition to my previous mail,
    below are the log messages, when job tracker does not start

    2012-12-02 06:36:26,190 INFO
    [CommandPusher:service.AbstractBringUpBringDownCommands@505] BringUp
    command (155) has finished unsuccessfully on service DbService{id=10,
    name=mapreduce1, serviceType=MAPREDUCE, optimisticLockVersion=8} role id:28
    name: mapreduce1-JOBTRACKER-2 hostId:ip-10-40-222-77.ec2.internal
    roleType:JOBTRACKER configuredStatus:STOPPED configGeneration:6 status:NA
    service:mapreduce1.


    2012-12-02 00:02:53,680 WARN org.apache.hadoop.mapred.JobTracker: Error
    starting tracker: ENOENT: No such file or directory
    at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
    at
    org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:521)
    at
    org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
    at
    org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
    at org.apache.hadoop.mapred.JobHistory.init(JobHistory.java:335)
    at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2178)
    at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2175)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2175)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2050)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:296)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:288)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4792)



    2012-12-02 00:02:54,739 FATAL org.apache.hadoop.mapred.JobTracker:
    java.net.BindException: Problem binding to ip-10-40-222-77.ec2.internal/
    10.40.222.77:8021 : Address already in use
    at org.apache.hadoop.ipc.Server.bind(Server.java:230)
    at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:319)
    at org.apache.hadoop.ipc.Server.<init>(Server.java:1529)
    at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:539)
    at org.apache.hadoop.ipc.RPC.getServer(RPC.java:500)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2143)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2050)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:296)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:288)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4792)
    Caused by: java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind(Native Method)
    at
    sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
    at
    sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
    at org.apache.hadoop.ipc.Server.bind(Server.java:228)
    ... 9 more

    2012-12-02 00:02:54,757 INFO org.apache.hadoop.mapred.JobTracker:
    SHUTDOWN_MSG:
    /************************************************************
    SHUTDOWN_MSG: Shutting down JobTracker at ip-10-40-222-77.ec2.internal/
    10.40.222.77


    Please help.

    Regards
    Shouvanik


    On Sun, Dec 2, 2012 at 4:20 PM, bc Wong wrote:
    On Sun, Dec 2, 2012 at 12:05 AM, Dalia Hassan wrote:

    I have been in this mess for a couple of days, when I try to solve a
    problem another occurs, don't know why.

    now when I start the zookeeper i get the following error although all
    hosts are up.

    Service did not start successfully; not all of the required roles
    started: No ZooKeeper servers are running.
    Dalia,

    I noticed your frustration from your choice of subject title. And I
    looked through the questions that you've posted in this mailing list. Most
    threads have recommendations but not the final resolution. But from what I
    can tell, you're having a fair amount of environmental issues (like DNS),
    especially if you're running in a cloud. While Cloudera Manager tries to
    eliminate the need to have special Hadoop knowledge and sysadmin expertise,
    we're not there yet. Running a distributed system is fundamentally hard.
    Everyone on the CM team works day and night to build solid user-friendly
    features. The software can cover more error cases, and can give most useful
    error messages, and we will make that happen.

    For this specific problem, it seems that your ZK servers are not
    starting right. Could you attach the ZK server log (go to CM -> ZK ->
    Server -> Processes -> Role log)? And make sure that the agents are
    heartbeating (go to CM -> Hosts).

    Thanks for your patience.

    Cheers,
    bc


    --
    Thanks,
    *Shouvanik*
  • Dilip Kumar at Aug 22, 2013 at 7:42 am
    Hi,

    I'm also getting same problem.where should i create the directory manually
    as automatic data directory creation is disabled.

    Thanks in advance

    On Monday, December 3, 2012 11:09:44 AM UTC+5:30, bc Wong wrote:

    On Sun, Dec 2, 2012 at 6:49 AM, Dalia Hassan <daliah...@gmail.com<javascript:>
    wrote:
    December 2 2012 4:15 PM Time Log Level Source Log Message 4:09:34.878
    PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Reading configuration from: /run/cloudera-scm-agent/process/168-zookeeper-server/zoo.cfg

    4:09:34.886 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Invalid configuration, only one server specified (ignoring)

    4:09:34.899 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    autopurge.snapRetainCount set to 5

    4:09:34.900 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    autopurge.purgeInterval set to 24

    4:09:34.902 PM WARN org.apache.zookeeper.server.quorum.QuorumPeerMain

    Either no config or no quorum defined in config, running in standalone mode

    4:09:34.909 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    Purge task started.

    4:09:34.953 PM ERROR org.apache.zookeeper.server.DatadirCleanupManager

    Error occured while purging.
    org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Missing data directory /var/lib/zookeeper/version-2, automatic data directory creation is disabled (zookeeper.datadir.autocreate is false). Please create this directory manually.
    at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:97)
    at org.apache.zookeeper.server.PurgeTxnLog.purge(PurgeTxnLog.java:68)
    at org.apache.zookeeper.server.DatadirCleanupManager$PurgeTask.run(DatadirCleanupManager.java:140)
    at java.util.TimerThread.mainLoop(Timer.java:512)
    at java.util.TimerThread.run(Timer.java:462)
    Dalia,

    This is a separate problem from the one Shouvanik posted (which was a
    port conflict on 8021 for the JT). Here, the "missing data directory" seems
    to be the issue.

    Go to CM. Make sure that the ZK service is stopped. (It's probably not
    running to begin with.) Then go to ZK -> Server -> Actions -> Initialize.
    If the command succeeds, then you can restart your ZK. Otherwise, can you
    attach the failure message for the "Initialize" command?

    Cheers,
    bc

    4:09:34.956 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    Purge task completed.

    4:09:34.990 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Reading configuration from: /run/cloudera-scm-agent/process/168-zookeeper-server/zoo.cfg

    4:09:34.991 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Invalid configuration, only one server specified (ignoring)

    4:09:34.992 PM INFO org.apache.zookeeper.server.ZooKeeperServerMain

    Starting server

    4:09:35.020 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:zookeeper.version=3.4.3-cdh4.1.2--1, built on 11/01/2012 23:59 GMT

    4:09:35.020 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:host.name=slave4.medcloud.com

    4:09:35.020 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.version=1.6.0_31

    4:09:35.021 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.vendor=Sun Microsystems Inc.

    4:09:35.021 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.home=/usr/lib/jvm/j2sdk1.6-oracle/jre

    4:09:35.021 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.class.path=/run/cloudera-scm-agent/process/168-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.4.3-cdh4.1.2.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar:/usr/lib/zookeeper/lib/netty-3.2.2.Final.jar:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar

    4:09:35.021 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.library.path=/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64/server:/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64:/usr/lib/jvm/j2sdk1.6-oracle/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

    4:09:35.021 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.io.tmpdir=/tmp

    4:09:35.021 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.compiler=<NA>

    4:09:35.021 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:os.name=Linux

    4:09:35.022 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:os.arch=amd64

    4:09:35.022 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:os.version=3.2.0-29-generic

    4:09:35.022 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:user.name=zookeeper

    4:09:35.022 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:user.home=/var/lib/zookeeper

    4:09:35.022 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:user.dir=/run/cloudera-scm-agent/process/168-zookeeper-server

    4:09:35.025 PM ERROR org.apache.zookeeper.server.ZooKeeperServerMain

    Unable to access datadir, exiting abnormally
    org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Missing data directory /var/lib/zookeeper/version-2, automatic data directory creation is disabled (zookeeper.datadir.autocreate is false). Please create this directory manually.
    at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:97)
    at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:108)
    at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:91)
    at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:53)
    at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:121)
    at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)

    4:09:36.939 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Reading configuration from: /run/cloudera-scm-agent/process/168-zookeeper-server/zoo.cfg

    4:09:36.945 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Invalid configuration, only one server specified (ignoring)

    4:09:36.948 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    autopurge.snapRetainCount set to 5

    4:09:36.948 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    autopurge.purgeInterval set to 24

    4:09:36.950 PM WARN org.apache.zookeeper.server.quorum.QuorumPeerMain

    Either no config or no quorum defined in config, running in standalone mode

    4:09:36.963 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    Purge task started.

    4:09:36.987 PM ERROR org.apache.zookeeper.server.DatadirCleanupManager

    Error occured while purging.
    org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Missing data directory /var/lib/zookeeper/version-2, automatic data directory creation is disabled (zookeeper.datadir.autocreate is false). Please create this directory manually.
    at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:97)
    at org.apache.zookeeper.server.PurgeTxnLog.purge(PurgeTxnLog.java:68)
    at org.apache.zookeeper.server.DatadirCleanupManager$PurgeTask.run(DatadirCleanupManager.java:140)
    at java.util.TimerThread.mainLoop(Timer.java:512)
    at java.util.TimerThread.run(Timer.java:462)

    4:09:36.990 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    Purge task completed.

    4:09:36.997 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Reading configuration from: /run/cloudera-scm-agent/process/168-zookeeper-server/zoo.cfg

    4:09:36.999 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Invalid configuration, only one server specified (ignoring)

    4:09:36.999 PM INFO org.apache.zookeeper.server.ZooKeeperServerMain

    Starting server

    4:09:37.029 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:zookeeper.version=3.4.3-cdh4.1.2--1, built on 11/01/2012 23:59 GMT

    4:09:37.030 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:host.name=slave4.medcloud.com

    4:09:37.030 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.version=1.6.0_31

    4:09:37.030 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.vendor=Sun Microsystems Inc.

    4:09:37.030 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.home=/usr/lib/jvm/j2sdk1.6-oracle/jre

    4:09:37.030 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.class.path=/run/cloudera-scm-agent/process/168-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.4.3-cdh4.1.2.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar:/usr/lib/zookeeper/lib/netty-3.2.2.Final.jar:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar

    4:09:37.030 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.library.path=/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64/server:/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64:/usr/lib/jvm/j2sdk1.6-oracle/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

    4:09:37.030 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.io.tmpdir=/tmp

    4:09:37.031 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.compiler=<NA>

    4:09:37.031 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:os.name=Linux

    4:09:37.031 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:os.arch=amd64

    4:09:37.031 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:os.version=3.2.0-29-generic

    4:09:37.031 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:user.name=zookeeper

    4:09:37.031 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:user.home=/var/lib/zookeeper

    4:09:37.031 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:user.dir=/run/cloudera-scm-agent/process/168-zookeeper-server

    4:09:37.033 PM ERROR org.apache.zookeeper.server.ZooKeeperServerMain

    Unable to access datadir, exiting abnormally
    org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Missing data directory /var/lib/zookeeper/version-2, automatic data directory creation is disabled (zookeeper.datadir.autocreate is false). Please create this directory manually.
    at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:97)
    at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:108)
    at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:91)
    at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:53)
    at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:121)
    at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)

    4:09:39.957 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Reading configuration from: /run/cloudera-scm-agent/process/168-zookeeper-server/zoo.cfg

    4:09:39.963 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Invalid configuration, only one server specified (ignoring)

    4:09:39.966 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    autopurge.snapRetainCount set to 5

    4:09:39.966 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    autopurge.purgeInterval set to 24

    4:09:39.968 PM WARN org.apache.zookeeper.server.quorum.QuorumPeerMain

    Either no config or no quorum defined in config, running in standalone mode

    4:09:39.975 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    Purge task started.

    4:09:40.005 PM ERROR org.apache.zookeeper.server.DatadirCleanupManager

    Error occured while purging.
    org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Missing data directory /var/lib/zookeeper/version-2, automatic data directory creation is disabled (zookeeper.datadir.autocreate is false). Please create this directory manually.
    at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:97)
    at org.apache.zookeeper.server.PurgeTxnLog.purge(PurgeTxnLog.java:68)
    at org.apache.zookeeper.server.DatadirCleanupManager$PurgeTask.run(DatadirCleanupManager.java:140)
    at java.util.TimerThread.mainLoop(Timer.java:512)
    at java.util.TimerThread.run(Timer.java:462)

    4:09:40.007 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    Purge task completed.

    4:09:40.027 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Reading configuration from: /run/cloudera-scm-agent/process/168-zookeeper-server/zoo.cfg

    4:09:40.028 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Invalid configuration, only one server specified (ignoring)

    4:09:40.028 PM INFO org.apache.zookeeper.server.ZooKeeperServerMain

    Starting server

    4:09:40.050 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:zookeeper.version=3.4.3-cdh4.1.2--1, built on 11/01/2012 23:59 GMT

    4:09:40.051 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:host.name=slave4.medcloud.com

    4:09:40.051 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.version=1.6.0_31

    4:09:40.051 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.vendor=Sun Microsystems Inc.

    4:09:40.051 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.home=/usr/lib/jvm/j2sdk1.6-oracle/jre

    4:09:40.051 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.class.path=/run/cloudera-scm-agent/process/168-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.4.3-cdh4.1.2.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar:/usr/lib/zookeeper/lib/netty-3.2.2.Final.jar:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar

    4:09:40.051 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.library.path=/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64/server:/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64:/usr/lib/jvm/j2sdk1.6-oracle/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

    4:09:40.051 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.io.tmpdir=/tmp

    4:09:40.051 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.compiler=<NA>

    4:09:40.052 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:os.name=Linux

    4:09:40.052 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:os.arch=amd64

    4:09:40.052 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:os.version=3.2.0-29-generic

    4:09:40.052 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:user.name=zookeeper

    4:09:40.052 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:user.home=/var/lib/zookeeper

    4:09:40.052 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:user.dir=/run/cloudera-scm-agent/process/168-zookeeper-server

    4:09:40.054 PM ERROR org.apache.zookeeper.server.ZooKeeperServerMain

    Unable to access datadir, exiting abnormally
    org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Missing data directory /var/lib/zookeeper/version-2, automatic data directory creation is disabled (zookeeper.datadir.autocreate is false). Please create this directory manually.
    at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:97)
    at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:108)
    at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:91)
    at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:53)
    at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:121)
    at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)

    4:09:43.969 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Reading configuration from: /run/cloudera-scm-agent/process/168-zookeeper-server/zoo.cfg

    4:09:43.975 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Invalid configuration, only one server specified (ignoring)

    4:09:43.978 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    autopurge.snapRetainCount set to 5

    4:09:43.978 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    autopurge.purgeInterval set to 24

    4:09:43.980 PM WARN org.apache.zookeeper.server.quorum.QuorumPeerMain

    Either no config or no quorum defined in config, running in standalone mode

    4:09:43.987 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    Purge task started.

    4:09:44.015 PM ERROR org.apache.zookeeper.server.DatadirCleanupManager

    Error occured while purging.
    org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Missing data directory /var/lib/zookeeper/version-2, automatic data directory creation is disabled (zookeeper.datadir.autocreate is false). Please create this directory manually.
    at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:97)
    at org.apache.zookeeper.server.PurgeTxnLog.purge(PurgeTxnLog.java:68)
    at org.apache.zookeeper.server.DatadirCleanupManager$PurgeTask.run(DatadirCleanupManager.java:140)
    at java.util.TimerThread.mainLoop(Timer.java:512)
    at java.util.TimerThread.run(Timer.java:462)

    4:09:44.017 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    Purge task completed.

    4:09:44.039 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Reading configuration from: /run/cloudera-scm-agent/process/168-zookeeper-server/zoo.cfg

    4:09:44.041 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Invalid configuration, only one server specified (ignoring)

    4:09:44.041 PM INFO org.apache.zookeeper.server.ZooKeeperServerMain

    Starting server

    4:09:44.063 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:zookeeper.version=3.4.3-cdh4.1.2--1, built on 11/01/2012 23:59 GMT

    4:09:44.063 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:host.name=slave4.medcloud.com

    4:09:44.064 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.version=1.6.0_31

    4:09:44.064 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.vendor=Sun Microsystems Inc.

    4:09:44.064 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.home=/usr/lib/jvm/j2sdk1.6-oracle/jre

    4:09:44.064 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.class.path=/run/cloudera-scm-agent/process/168-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.4.3-cdh4.1.2.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar:/usr/lib/zookeeper/lib/netty-3.2.2.Final.jar:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar

    4:09:44.064 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.library.path=/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64/server:/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64:/usr/lib/jvm/j2sdk1.6-oracle/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

    4:09:44.064 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.io.tmpdir=/tmp

    4:09:44.064 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.compiler=<NA>

    4:09:44.064 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:os.name=Linux

    4:09:44.064 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:os.arch=amd64

    4:09:44.065 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:os.version=3.2.0-29-generic

    4:09:44.065 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:user.name=zookeeper

    4:09:44.065 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:user.home=/var/lib/zookeeper

    4:09:44.065 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:user.dir=/run/cloudera-scm-agent/process/168-zookeeper-server

    4:09:44.067 PM ERROR org.apache.zookeeper.server.ZooKeeperServerMain

    Unable to access datadir, exiting abnormally
    org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Missing data directory /var/lib/zookeeper/version-2, automatic data directory creation is disabled (zookeeper.datadir.autocreate is false). Please create this directory manually.
    at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:97)
    at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:108)
    at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:91)
    at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:53)
    at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:121)
    at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)

    4:10:18.647 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Reading configuration from: /run/cloudera-scm-agent/process/170-zookeeper-server/zoo.cfg

    4:10:18.653 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Invalid configuration, only one server specified (ignoring)

    4:10:18.656 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    autopurge.snapRetainCount set to 5

    4:10:18.656 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    autopurge.purgeInterval set to 24

    4:10:18.658 PM WARN org.apache.zookeeper.server.quorum.QuorumPeerMain

    Either no config or no quorum defined in config, running in standalone mode

    4:10:18.667 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    Purge task started.

    4:10:18.717 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    Purge task completed.

    4:10:18.726 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Reading configuration from: /run/cloudera-scm-agent/process/170-zookeeper-server/zoo.cfg

    4:10:18.727 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Invalid configuration, only one server specified (ignoring)

    4:10:18.727 PM INFO org.apache.zookeeper.server.ZooKeeperServerMain

    Starting server

    4:10:18.748 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:zookeeper.version=3.4.3-cdh4.1.2--1, built on 11/01/2012 23:59 GMT

    4:10:18.748 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:host.name=slave4.medcloud.com

    4:10:18.748 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.version=1.6.0_31

    4:10:18.748 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.vendor=Sun Microsystems Inc.

    4:10:18.749 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.home=/usr/lib/jvm/j2sdk1.6-oracle/jre

    4:10:18.749 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.class.path=/run/cloudera-scm-agent/process/170-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.4.3-cdh4.1.2.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar:/usr/lib/zookeeper/lib/netty-3.2.2.Final.jar:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar

    4:10:18.749 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.library.path=/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64/server:/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64:/usr/lib/jvm/j2sdk1.6-oracle/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

    4:10:18.749 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.io.tmpdir=/tmp

    4:10:18.749 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:java.compiler=<NA>

    4:10:18.749 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:os.name=Linux

    4:10:18.749 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:os.arch=amd64

    4:10:18.749 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:os.version=3.2.0-29-generic

    4:10:18.749 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:user.name=zookeeper

    4:10:18.750 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:user.home=/var/lib/zookeeper

    4:10:18.750 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Server environment:user.dir=/run/cloudera-scm-agent/process/170-zookeeper-server

    4:10:18.752 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    tickTime set to 2000

    4:10:18.752 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    minSessionTimeout set to 4000

    4:10:18.752 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    maxSessionTimeout set to 40000

    4:10:18.838 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

    binding to port 0.0.0.0/0.0.0.0:2181

    4:10:18.888 PM INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog

    Snapshotting: 0x0 to /var/lib/zookeeper/version-2/snapshot.0

    4:14:10.689 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

    Accepted socket connection from /192.168.0.8:39761

    4:14:10.738 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Client attempting to establish new session at /192.168.0.8:39761

    4:14:10.764 PM INFO org.apache.zookeeper.server.persistence.FileTxnLog

    Creating new log file: log.1

    4:14:10.868 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Established session 0x13b5bf40c940000 with negotiated timeout 40000 for client /192.168.0.8:39761

    4:14:11.028 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

    Accepted socket connection from /192.168.0.5:35186

    4:14:11.041 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Client attempting to establish new session at /192.168.0.5:35186

    4:14:11.050 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Established session 0x13b5bf40c940001 with negotiated timeout 40000 for client /192.168.0.5:35186

    4:14:11.161 PM INFO org.apache.zookeeper.server.PrepRequestProcessor

    Got user-level KeeperException when processing sessionid:0x13b5bf40c940000 type:delete cxid:0x11 zxid:0xb txntype:-1 reqpath:n/a Error Path:/hbase/backup-masters/slave7.medcloud.com,60000,1354457649683 Error:KeeperErrorCode = NoNode for /hbase/backup-masters/slave7.medcloud.com,60000,1354457649683

    4:14:12.589 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

    Accepted socket connection from /192.168.0.4:48929

    4:14:12.618 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Client attempting to establish new session at /192.168.0.4:48929

    4:14:12.633 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Established session 0x13b5bf40c940002 with negotiated timeout 40000 for client /192.168.0.4:48929

    4:14:13.654 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

    Accepted socket connection from /192.168.0.2:59878

    4:14:13.667 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Client attempting to establish new session at /192.168.0.2:59878

    4:14:13.683 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Established session 0x13b5bf40c940003 with negotiated timeout 40000 for client /192.168.0.2:59878

    4:14:14.165 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

    Accepted socket connection from /192.168.0.8:39764

    4:14:14.171 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Client attempting to establish new session at /192.168.0.8:39764

    4:14:14.175 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Established session 0x13b5bf40c940004 with negotiated timeout 40000 for client /192.168.0.8:39764

    4:14:14.297 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

    Accepted socket connection from /192.168.0.5:35187

    4:14:14.299 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Client attempting to establish new session at /192.168.0.5:35187

    4:14:14.309 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Established session 0x13b5bf40c940005 with negotiated timeout 40000 for client /192.168.0.5:35187

    4:14:14.310 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

    Accepted socket connection from /192.168.0.4:48930

    4:14:14.311 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Client attempting to establish new session at /192.168.0.4:48930

    4:14:14.316 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Established session 0x13b5bf40c940006 with negotiated timeout 40000 for client /192.168.0.4:48930

    4:14:14.543 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

    Accepted socket connection from /192.168.0.2:59879

    4:14:14.544 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Client attempting to establish new session at /192.168.0.2:59879

    4:14:14.558 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Established session 0x13b5bf40c940007 with negotiated timeout 40000 for client /192.168.0.2:59879

    4:14:14.563 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

    Accepted socket connection from /192.168.0.3:47913

    4:14:14.588 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Client attempting to establish new session at /192.168.0.3:47913

    4:14:14.591 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Established session 0x13b5bf40c940008 with negotiated timeout 40000 for client /192.168.0.3:47913

    4:14:14.806 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

    Accepted socket connection from /192.168.0.3:47914

    4:14:14.807 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Client attempting to establish new session at /192.168.0.3:47914

    4:14:14.816 PM INFO org.apache.zookeeper.server.ZooKeeperServer

    Established session 0x13b5bf40c940009 with negotiated timeout 40000 for client /192.168.0.3:47914

    4:14:21.841 PM INFO org.apache.zookeeper.server.PrepRequestProcessor

    Got user-level KeeperException when processing sessionid:0x13b5bf40c940000 type:delete cxid:0x30 zxid:0x1b txntype:-1 reqpath:n/a Error Path:/hbase/root-region-server Error:KeeperErrorCode = NoNode for /hbase/root-region-server

    4:14:41.679 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Reading configuration from: /run/cloudera-scm-agent/process/183-zookeeper-server/zoo.cfg

    4:14:41.685 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

    Invalid configuration, only one server specified (ignoring)

    4:14:41.688 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    autopurge.snapRetainCount set to 5

    4:14:41.688 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    autopurge.purgeInterval set to 24

    4:14:41.690 PM WARN org.apache.zookeeper.server.quorum.QuorumPeerMain

    Either no config or no quorum defined in config, running in standalone mode

    4:14:41.699 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

    Purge task started.



    On Sun, Dec 2, 2012 at 1:41 PM, Shouvanik Haldar <shouvani...@gmail.com<javascript:>
    wrote:
    In addition to my previous mail,
    below are the log messages, when job tracker does not start

    2012-12-02 06:36:26,190 INFO
    [CommandPusher:service.AbstractBringUpBringDownCommands@505] BringUp
    command (155) has finished unsuccessfully on service DbService{id=10,
    name=mapreduce1, serviceType=MAPREDUCE, optimisticLockVersion=8} role id:28
    name: mapreduce1-JOBTRACKER-2 hostId:ip-10-40-222-77.ec2.internal
    roleType:JOBTRACKER configuredStatus:STOPPED configGeneration:6 status:NA
    service:mapreduce1.


    2012-12-02 00:02:53,680 WARN org.apache.hadoop.mapred.JobTracker: Error
    starting tracker: ENOENT: No such file or directory
    at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
    at
    org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:521)
    at
    org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
    at
    org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
    at org.apache.hadoop.mapred.JobHistory.init(JobHistory.java:335)
    at
    org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2178)
    at
    org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2175)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2175)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2050)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:296)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:288)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4792)



    2012-12-02 00:02:54,739 FATAL org.apache.hadoop.mapred.JobTracker:
    java.net.BindException: Problem binding to ip-10-40-222-77.ec2.internal/
    10.40.222.77:8021 : Address already in use
    at org.apache.hadoop.ipc.Server.bind(Server.java:230)
    at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:319)
    at org.apache.hadoop.ipc.Server.<init>(Server.java:1529)
    at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:539)
    at org.apache.hadoop.ipc.RPC.getServer(RPC.java:500)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2143)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2050)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:296)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:288)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4792)
    Caused by: java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind(Native Method)
    at
    sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
    at
    sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
    at org.apache.hadoop.ipc.Server.bind(Server.java:228)
    ... 9 more

    2012-12-02 00:02:54,757 INFO org.apache.hadoop.mapred.JobTracker:
    SHUTDOWN_MSG:
    /************************************************************
    SHUTDOWN_MSG: Shutting down JobTracker at ip-10-40-222-77.ec2.internal/
    10.40.222.77


    Please help.

    Regards
    Shouvanik



    On Sun, Dec 2, 2012 at 4:20 PM, bc Wong <bcwa...@cloudera.com<javascript:>
    wrote:
    On Sun, Dec 2, 2012 at 12:05 AM, Dalia Hassan <daliah...@gmail.com<javascript:>
    wrote:
    I have been in this mess for a couple of days, when I try to solve a
    problem another occurs, don't know why.

    now when I start the zookeeper i get the following error although all
    hosts are up.

    Service did not start successfully; not all of the required roles
    started: No ZooKeeper servers are running.
    Dalia,

    I noticed your frustration from your choice of subject title. And I
    looked through the questions that you've posted in this mailing list. Most
    threads have recommendations but not the final resolution. But from what I
    can tell, you're having a fair amount of environmental issues (like DNS),
    especially if you're running in a cloud. While Cloudera Manager tries to
    eliminate the need to have special Hadoop knowledge and sysadmin expertise,
    we're not there yet. Running a distributed system is fundamentally hard.
    Everyone on the CM team works day and night to build solid user-friendly
    features. The software can cover more error cases, and can give most useful
    error messages, and we will make that happen.

    For this specific problem, it seems that your ZK servers are not
    starting right. Could you attach the ZK server log (go to CM -> ZK ->
    Server -> Processes -> Role log)? And make sure that the agents are
    heartbeating (go to CM -> Hosts).

    Thanks for your patience.

    Cheers,
    bc


    --
    Thanks,
    *Shouvanik*
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedDec 2, '12 at 2:49p
activeAug 22, '13 at 7:42a
posts3
users3
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase