FAQ
Hi,

I installed Cloudera Manager(4.1.1) and CDH4 (4.1.2) using tarball on
Ubuntu 10.04.4 (VM using virtual box). I was able to setup Cloudera
manager server and agent, setup roles (NN,DN and Secondary NN) in pseudo
distributed mode. I deployed the configuration to hosts and formatted name
node successfully.

When I started the services from Cloudera manager, I got message saying all
3 (NN,DN,SNN) were started successfully, but I cannot access anyone of them
through web url.
The status for all 3 is started with good health.


I looked at the log file for namenode and it is stuck at this line


"2012-12-02 10:33:33,841 INFO
org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Going to
retain 2 images with txid >= 0"

This is the first start after format, so I am confused why this is
happening... Any help would be really appreciated..


Thanks
JH





FULL LOG FILE: ***************************



************************************************************/
2012-12-02 10:32:19,931 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = ubuntu1/127.0.0.1
STARTUP_MSG: args = []
STARTUP_MSG: version = 2.0.0-cdh4.1.2
STARTUP_MSG: classpath =
/home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-log4j12-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-json-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jettison-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jets3t-0.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-digester-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-annotations-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/activation-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/kfs-0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-net-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-auth-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsch-0.1.42.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-api-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/paranamer-2.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-math-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/junit-4.8.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/avro-1.7.1.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/stax-api-1.0.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-tests.jar:/home/john/cloudera-manager/cm-4.1.1/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-daemon-1.0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-tests.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-sources.jar:/usr/lib/hadoop-yarn/share/hadoop/yarn/*:/usr/lib/hadoop-mapreduce/share/hadoop/mapreduce/*
STARTUP_MSG: build =
file:///var/lib/jenkins/workspace/CDH4.1.2-Packaging-Hadoop/build/cdh4/hadoop/2.0.0-cdh4.1.2/source/hadoop-common-project/hadoop-common-r f0b53c81cbf56f5955e403b49fcd27afd5f082de; compiled by 'jenkins' on Thu
Nov 1 17:06:55 PDT 2012
************************************************************/
2012-12-02 10:32:22,214 INFO org.apache.hadoop.metrics2.impl.MetricsConfig:
loaded properties from hadoop-metrics2.properties
2012-12-02 10:32:23,240 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
period at 10 second(s).
2012-12-02 10:32:23,240 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
started
2012-12-02 10:32:25,269 INFO org.apache.hadoop.util.HostsFileReader:
Refreshing hosts (include/exclude) list
2012-12-02 10:32:25,270 INFO org.apache.hadoop.util.HostsFileReader: Adding
127.0.0.1 to the list of hosts from
/home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE/dfs_hosts_allow.txt
2012-12-02 10:32:25,362 INFO
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
dfs.block.invalidate.limit=1000
2012-12-02 10:32:25,422 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
dfs.block.access.token.enable=false
2012-12-02 10:32:25,441 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
defaultReplication = 1
2012-12-02 10:32:25,441 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
maxReplication = 512
2012-12-02 10:32:25,441 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
minReplication = 1
2012-12-02 10:32:25,441 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
maxReplicationStreams = 2
2012-12-02 10:32:25,441 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
shouldCheckForEnoughRacks = true
2012-12-02 10:32:25,441 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
replicationRecheckInterval = 3000
2012-12-02 10:32:25,441 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
encryptDataTransfer = false
2012-12-02 10:32:25,454 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
john (auth:SIMPLE)
2012-12-02 10:32:25,454 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
john
2012-12-02 10:32:25,454 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
true
2012-12-02 10:32:25,454 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2012-12-02 10:32:25,480 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2012-12-02 10:32:27,741 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
occuring more than 10 times
2012-12-02 10:32:27,743 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2012-12-02 10:32:27,743 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.min.datanodes = 0
2012-12-02 10:32:27,743 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.extension = 30000
2012-12-02 10:32:27,941 INFO org.apache.hadoop.hdfs.server.common.Storage:
Lock on /home/john/dfs_dir/namenode/data1/in_use.lock acquired by nodename
2070@ubuntu1
2012-12-02 10:32:28,574 INFO org.apache.hadoop.hdfs.server.common.Storage:
Lock on /home/john/dfs_dir/namenode/data2/in_use.lock acquired by nodename
2070@ubuntu1
2012-12-02 10:32:28,803 INFO
org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering
unfinalized segments in /home/john/dfs_dir/namenode/data1/current
2012-12-02 10:32:28,803 INFO
org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering
unfinalized segments in /home/john/dfs_dir/namenode/data2/current
2012-12-02 10:32:28,805 INFO
org.apache.hadoop.hdfs.server.namenode.FSImage: No edit log streams
selected.
2012-12-02 10:32:28,950 INFO
org.apache.hadoop.hdfs.server.namenode.FSImage: Loading image file
/home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000 using
no compression
2012-12-02 10:32:28,950 INFO
org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files = 1
2012-12-02 10:32:28,951 INFO
org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files under
construction = 0
2012-12-02 10:32:28,952 INFO
org.apache.hadoop.hdfs.server.namenode.FSImage: Image file of size 117
loaded in 0 seconds.
2012-12-02 10:32:28,952 INFO
org.apache.hadoop.hdfs.server.namenode.FSImage: Loaded image for txid 0
from /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000
2012-12-02 10:32:28,955 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 1
2012-12-02 10:32:30,041 INFO
org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with 0
entries 0 lookups
2012-12-02 10:32:30,041 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading
FSImage in 2298 msecs
2012-12-02 10:32:31,360 INFO org.apache.hadoop.ipc.Server: Starting Socket
Reader #1 for port 8020
2012-12-02 10:32:31,402 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
FSNamesystemState MBean
2012-12-02 10:32:31,482 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under
construction: 0
2012-12-02 10:32:31,482 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: initializing
replication queues
2012-12-02 10:32:31,513 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Total number of
blocks = 0
2012-12-02 10:32:31,513 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of
invalid blocks = 0
2012-12-02 10:32:31,513 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of
under-replicated blocks = 0
2012-12-02 10:32:31,513 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of
over-replicated blocks = 0
2012-12-02 10:32:31,513 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of
blocks being written = 0
2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange: STATE*
Replication Queue initialization scan for invalid, over- and
under-replicated blocks completed in 31 msec
2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange: STATE*
Leaving safe mode after 6 secs.
2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange: STATE*
Network topology has 0 racks and 0 datanodes
2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange: STATE*
UnderReplicatedBlocks has 0 blocks
2012-12-02 10:32:31,674 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2012-12-02 10:32:31,814 INFO org.apache.hadoop.http.HttpServer: Added
global filter 'safety'
(class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer: Added
filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context hdfs
2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer: Added
filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context logs
2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer: Added
filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context static
2012-12-02 10:32:31,841 INFO org.apache.hadoop.http.HttpServer:
dfs.webhdfs.enabled = true
2012-12-02 10:32:31,842 INFO org.apache.hadoop.http.HttpServer: Added
filter 'SPNEGO' (class=org.apache.hadoop.hdfs.web.AuthFilter)
2012-12-02 10:32:31,844 INFO org.apache.hadoop.http.HttpServer:
addJerseyResourcePackage:
packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
pathSpec=/webhdfs/v1/*
2012-12-02 10:32:31,862 INFO org.apache.hadoop.http.HttpServer: Jetty bound
to port 50070
2012-12-02 10:32:31,862 INFO org.mortbay.log: jetty-6.1.26.cloudera.2
2012-12-02 10:32:32,261 WARN
org.apache.hadoop.security.authentication.server.AuthenticationFilter:
'signature.secret' configuration not set, using a random value as secret
2012-12-02 10:32:32,453 INFO org.mortbay.log: Started
selectchannelconnector@localhost.localdomain:50070
2012-12-02 10:32:32,453 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at:
localhost.localdomain:50070
2012-12-02 10:32:32,453 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2012-12-02 10:32:32,454 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 8020: starting
2012-12-02 10:32:32,603 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode up at:
ubuntu1/127.0.0.1:8020
2012-12-02 10:32:32,603 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services
required for active state
2012-12-02 10:32:34,359 INFO org.apache.hadoop.fs.TrashPolicyDefault: The
configured checkpoint interval is 0 minutes. Using an interval of 1440
minutes that is used for deletion instead
2012-12-02 10:32:35,780 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.registerDatanode: node registration from
DatanodeRegistration(127.0.0.1,
storageID=DS-377461142-127.0.0.1-50010-1354473155123, infoPort=50075,
ipcPort=50020, storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0) storage
DS-377461142-127.0.0.1-50010-1354473155123
2012-12-02 10:32:35,790 INFO org.apache.hadoop.net.NetworkTopology: Adding
a new node: /default/127.0.0.1:50010
2012-12-02 10:32:35,951 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: BLOCK*
processReport: Received first block report from 127.0.0.1:50010 after
becoming active. Its block contents are no longer considered stale.
2012-12-02 10:32:35,951 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
processReport: from DatanodeRegistration(127.0.0.1,
storageID=DS-377461142-127.0.0.1-50010-1354473155123, infoPort=50075,
ipcPort=50020, storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0), blocks:
0, processing time: 1 msecs
2012-12-02 10:33:31,480 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
127.0.0.1
2012-12-02 10:33:31,480 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs.
2012-12-02 10:33:31,480 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 1
2012-12-02 10:33:31,480 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
Total time for transactions(ms): 1Number of transactions batched in Syncs:
0 Number of syncs: 2 SyncTimes(ms): 166 189
2012-12-02 10:33:31,595 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
Total time for transactions(ms): 1Number of transactions batched in Syncs:
0 Number of syncs: 3 SyncTimes(ms): 215 255
2012-12-02 10:33:31,603 INFO
org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
file
/home/john/dfs_dir/namenode/data1/current/edits_inprogress_0000000000000000001
->
/home/john/dfs_dir/namenode/data1/current/edits_0000000000000000001-0000000000000000002
2012-12-02 10:33:31,604 INFO
org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
file
/home/john/dfs_dir/namenode/data2/current/edits_inprogress_0000000000000000001
->
/home/john/dfs_dir/namenode/data2/current/edits_0000000000000000001-0000000000000000002
2012-12-02 10:33:31,604 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 3
2012-12-02 10:33:33,274 INFO
org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Opening connection
to
http://ubuntu1:50090/getimage?getimage=1&txid=2&storageInfo=-40:829593698:0:cluster2
2012-12-02 10:33:33,622 INFO
org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Transfer took 0.34s
at 0.00 KB/s
2012-12-02 10:33:33,622 INFO
org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Downloaded file
fsimage.ckpt_0000000000000000002 size 117 bytes.
2012-12-02 10:33:33,841 INFO
org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Going to
retain 2 images with txid >= 0

Search Discussions

  • Harsh J at Dec 2, 2012 at 7:55 pm
    I replied to this over your duplicate post at
    https://groups.google.com/a/cloudera.org/d/topic/cdh-user/ft02DH52BZ4/discussion
    On Mon, Dec 3, 2012 at 12:59 AM, John H wrote:

    Hi,

    I installed Cloudera Manager(4.1.1) and CDH4 (4.1.2) using tarball on
    Ubuntu 10.04.4 (VM using virtual box). I was able to setup Cloudera manager
    server and agent, setup roles (NN,DN and Secondary NN) in pseudo distributed
    mode. I deployed the configuration to hosts and formatted name node
    successfully.

    When I started the services from Cloudera manager, I got message saying all
    3 (NN,DN,SNN) were started successfully, but I cannot access anyone of them
    through web url.
    The status for all 3 is started with good health.


    I looked at the log file for namenode and it is stuck at this line



    "2012-12-02 10:33:33,841 INFO
    org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Going to
    retain 2 images with txid >= 0"

    This is the first start after format, so I am confused why this is
    happening... Any help would be really appreciated..


    Thanks
    JH





    FULL LOG FILE: ***************************



    ************************************************************/
    2012-12-02 10:32:19,931 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting NameNode
    STARTUP_MSG: host = ubuntu1/127.0.0.1
    STARTUP_MSG: args = []
    STARTUP_MSG: version = 2.0.0-cdh4.1.2
    STARTUP_MSG: classpath =
    /home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-log4j12-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-json-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jettison-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jets3t-0.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-digester-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-annotations-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/activation-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/kfs-0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-net-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-auth-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsch-0.1.42.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-api-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/paranamer-2.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-math-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/junit-4.8.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/avro-1.7.1.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/stax-api-1.0.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-tests.jar:/home/john/cloudera-manager/cm-4.1.1/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-daemon-1.0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-tests.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-sources.jar:/usr/lib/hadoop-yarn/share/hadoop/yarn/*:/usr/lib/hadoop-mapreduce/share/hadoop/mapreduce/*
    STARTUP_MSG: build =
    file:///var/lib/jenkins/workspace/CDH4.1.2-Packaging-Hadoop/build/cdh4/hadoop/2.0.0-cdh4.1.2/source/hadoop-common-project/hadoop-common
    -r f0b53c81cbf56f5955e403b49fcd27afd5f082de; compiled by 'jenkins' on Thu
    Nov 1 17:06:55 PDT 2012
    ************************************************************/
    2012-12-02 10:32:22,214 INFO org.apache.hadoop.metrics2.impl.MetricsConfig:
    loaded properties from hadoop-metrics2.properties
    2012-12-02 10:32:23,240 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period
    at 10 second(s).
    2012-12-02 10:32:23,240 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
    started
    2012-12-02 10:32:25,269 INFO org.apache.hadoop.util.HostsFileReader:
    Refreshing hosts (include/exclude) list
    2012-12-02 10:32:25,270 INFO org.apache.hadoop.util.HostsFileReader: Adding
    127.0.0.1 to the list of hosts from
    /home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE/dfs_hosts_allow.txt
    2012-12-02 10:32:25,362 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
    dfs.block.invalidate.limit=1000
    2012-12-02 10:32:25,422 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    dfs.block.access.token.enable=false
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    defaultReplication = 1
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
    = 512
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
    = 1
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    maxReplicationStreams = 2
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    shouldCheckForEnoughRacks = true
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    replicationRecheckInterval = 3000
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    encryptDataTransfer = false
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
    john (auth:SIMPLE)
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
    john
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
    true
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
    2012-12-02 10:32:25,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
    2012-12-02 10:32:27,741 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring
    more than 10 times
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.threshold-pct = 0.9990000128746033
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.min.datanodes = 0
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.extension = 30000
    2012-12-02 10:32:27,941 INFO org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /home/john/dfs_dir/namenode/data1/in_use.lock acquired by nodename
    2070@ubuntu1
    2012-12-02 10:32:28,574 INFO org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /home/john/dfs_dir/namenode/data2/in_use.lock acquired by nodename
    2070@ubuntu1
    2012-12-02 10:32:28,803 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering
    unfinalized segments in /home/john/dfs_dir/namenode/data1/current
    2012-12-02 10:32:28,803 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering
    unfinalized segments in /home/john/dfs_dir/namenode/data2/current
    2012-12-02 10:32:28,805 INFO org.apache.hadoop.hdfs.server.namenode.FSImage:
    No edit log streams selected.
    2012-12-02 10:32:28,950 INFO org.apache.hadoop.hdfs.server.namenode.FSImage:
    Loading image file
    /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000 using
    no compression
    2012-12-02 10:32:28,950 INFO org.apache.hadoop.hdfs.server.namenode.FSImage:
    Number of files = 1
    2012-12-02 10:32:28,951 INFO org.apache.hadoop.hdfs.server.namenode.FSImage:
    Number of files under construction = 0
    2012-12-02 10:32:28,952 INFO org.apache.hadoop.hdfs.server.namenode.FSImage:
    Image file of size 117 loaded in 0 seconds.
    2012-12-02 10:32:28,952 INFO org.apache.hadoop.hdfs.server.namenode.FSImage:
    Loaded image for txid 0 from
    /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000
    2012-12-02 10:32:28,955 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 1
    2012-12-02 10:32:30,041 INFO
    org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with 0 entries
    0 lookups
    2012-12-02 10:32:30,041 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading
    FSImage in 2298 msecs
    2012-12-02 10:32:31,360 INFO org.apache.hadoop.ipc.Server: Starting Socket
    Reader #1 for port 8020
    2012-12-02 10:32:31,402 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
    FSNamesystemState MBean
    2012-12-02 10:32:31,482 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under
    construction: 0
    2012-12-02 10:32:31,482 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: initializing
    replication queues
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Total number of
    blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of
    invalid blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of
    under-replicated blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of
    over-replicated blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of blocks
    being written = 0
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange: STATE*
    Replication Queue initialization scan for invalid, over- and
    under-replicated blocks completed in 31 msec
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange: STATE*
    Leaving safe mode after 6 secs.
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange: STATE*
    Network topology has 0 racks and 0 datanodes
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange: STATE*
    UnderReplicatedBlocks has 0 blocks
    2012-12-02 10:32:31,674 INFO org.mortbay.log: Logging to
    org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
    org.mortbay.log.Slf4jLog
    2012-12-02 10:32:31,814 INFO org.apache.hadoop.http.HttpServer: Added global
    filter 'safety' (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer: Added filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
    context hdfs
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer: Added filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
    context logs
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer: Added filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
    context static
    2012-12-02 10:32:31,841 INFO org.apache.hadoop.http.HttpServer:
    dfs.webhdfs.enabled = true
    2012-12-02 10:32:31,842 INFO org.apache.hadoop.http.HttpServer: Added filter
    'SPNEGO' (class=org.apache.hadoop.hdfs.web.AuthFilter)
    2012-12-02 10:32:31,844 INFO org.apache.hadoop.http.HttpServer:
    addJerseyResourcePackage:
    packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
    pathSpec=/webhdfs/v1/*
    2012-12-02 10:32:31,862 INFO org.apache.hadoop.http.HttpServer: Jetty bound
    to port 50070
    2012-12-02 10:32:31,862 INFO org.mortbay.log: jetty-6.1.26.cloudera.2
    2012-12-02 10:32:32,261 WARN
    org.apache.hadoop.security.authentication.server.AuthenticationFilter:
    'signature.secret' configuration not set, using a random value as secret
    2012-12-02 10:32:32,453 INFO org.mortbay.log: Started
    selectchannelconnector@localhost.localdomain:50070
    2012-12-02 10:32:32,453 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at:
    localhost.localdomain:50070
    2012-12-02 10:32:32,453 INFO org.apache.hadoop.ipc.Server: IPC Server
    Responder: starting
    2012-12-02 10:32:32,454 INFO org.apache.hadoop.ipc.Server: IPC Server
    listener on 8020: starting
    2012-12-02 10:32:32,603 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode up at:
    ubuntu1/127.0.0.1:8020
    2012-12-02 10:32:32,603 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services
    required for active state
    2012-12-02 10:32:34,359 INFO org.apache.hadoop.fs.TrashPolicyDefault: The
    configured checkpoint interval is 0 minutes. Using an interval of 1440
    minutes that is used for deletion instead
    2012-12-02 10:32:35,780 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
    NameSystem.registerDatanode: node registration from
    DatanodeRegistration(127.0.0.1,
    storageID=DS-377461142-127.0.0.1-50010-1354473155123, infoPort=50075,
    ipcPort=50020, storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0) storage
    DS-377461142-127.0.0.1-50010-1354473155123
    2012-12-02 10:32:35,790 INFO org.apache.hadoop.net.NetworkTopology: Adding a
    new node: /default/127.0.0.1:50010
    2012-12-02 10:32:35,951 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: BLOCK*
    processReport: Received first block report from 127.0.0.1:50010 after
    becoming active. Its block contents are no longer considered stale.
    2012-12-02 10:32:35,951 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
    processReport: from DatanodeRegistration(127.0.0.1,
    storageID=DS-377461142-127.0.0.1-50010-1354473155123, infoPort=50075,
    ipcPort=50020, storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0), blocks:
    0, processing time: 1 msecs
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
    127.0.0.1
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs.
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 1
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
    Total time for transactions(ms): 1Number of transactions batched in Syncs: 0
    Number of syncs: 2 SyncTimes(ms): 166 189
    2012-12-02 10:33:31,595 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
    Total time for transactions(ms): 1Number of transactions batched in Syncs: 0
    Number of syncs: 3 SyncTimes(ms): 215 255
    2012-12-02 10:33:31,603 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
    file
    /home/john/dfs_dir/namenode/data1/current/edits_inprogress_0000000000000000001
    ->
    /home/john/dfs_dir/namenode/data1/current/edits_0000000000000000001-0000000000000000002
    2012-12-02 10:33:31,604 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
    file
    /home/john/dfs_dir/namenode/data2/current/edits_inprogress_0000000000000000001
    ->
    /home/john/dfs_dir/namenode/data2/current/edits_0000000000000000001-0000000000000000002
    2012-12-02 10:33:31,604 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 3
    2012-12-02 10:33:33,274 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Opening connection
    to
    http://ubuntu1:50090/getimage?getimage=1&txid=2&storageInfo=-40:829593698:0:cluster2
    2012-12-02 10:33:33,622 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Transfer took 0.34s
    at 0.00 KB/s
    2012-12-02 10:33:33,622 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Downloaded file
    fsimage.ckpt_0000000000000000002 size 117 bytes.
    2012-12-02 10:33:33,841 INFO
    org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Going to
    retain 2 images with txid >= 0



    --
    Harsh J
  • Shouvanik Haldar at Dec 2, 2012 at 8:12 pm
    Hi Harsh,

    I am not able to start hbase although hbase service is up. I am getting
    weird zookeeper exception. can you please help

    [root@ip-10-40-222-77 ~]# tail -f
    /var/log/zookeeper/zookeeper-cmf-zookeeper1-SERVER-ip-10-40-222-77.ec2.internal.log
    2012-12-02 03:42:47,991 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
    KeeperException when processing sessionid:0x13b5ac2ad7a0002 type:create
    cxid:0x6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
    Path:/hbase/master Error:KeeperErrorCode = NodeExists for /hbase/master
    2012-12-02 03:42:48,001 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30002, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30003, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30001, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30002
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30003
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30001
    2012-12-02 03:42:48,182 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44800
    2012-12-02 03:42:48,194 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44800
    2012-12-02 03:42:48,196 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0003 with negotiated timeout 40000 for
    client /10.40.222.77:44800


    Regards,
    Shouvanik Haldar

    On Mon, Dec 3, 2012 at 1:24 AM, Harsh J wrote:

    I replied to this over your duplicate post at

    https://groups.google.com/a/cloudera.org/d/topic/cdh-user/ft02DH52BZ4/discussion
    On Mon, Dec 3, 2012 at 12:59 AM, John H wrote:

    Hi,

    I installed Cloudera Manager(4.1.1) and CDH4 (4.1.2) using tarball on
    Ubuntu 10.04.4 (VM using virtual box). I was able to setup Cloudera manager
    server and agent, setup roles (NN,DN and Secondary NN) in pseudo
    distributed
    mode. I deployed the configuration to hosts and formatted name node
    successfully.

    When I started the services from Cloudera manager, I got message saying all
    3 (NN,DN,SNN) were started successfully, but I cannot access anyone of them
    through web url.
    The status for all 3 is started with good health.


    I looked at the log file for namenode and it is stuck at this line



    "2012-12-02 10:33:33,841 INFO
    org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Going to
    retain 2 images with txid >= 0"

    This is the first start after format, so I am confused why this is
    happening... Any help would be really appreciated..


    Thanks
    JH





    FULL LOG FILE: ***************************



    ************************************************************/
    2012-12-02 10:32:19,931 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting NameNode
    STARTUP_MSG: host = ubuntu1/127.0.0.1
    STARTUP_MSG: args = []
    STARTUP_MSG: version = 2.0.0-cdh4.1.2
    STARTUP_MSG: classpath =
    /home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-log4j12-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-json-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jettison-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jets3t-0.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-digester-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-annotations-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/activation-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/kfs-0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-net-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-auth-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsch-0.1.42.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-api-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/paranamer-2.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-math-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/junit-4.8.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/avro-1.7.1.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/stax-api-1.0.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-tests.jar:/home/john/cloudera-manager/cm-4.1.1/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-daemon-1.0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-tests.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-sources.jar:/usr/lib/hadoop-yarn/share/hadoop/yarn/*:/usr/lib/hadoop-mapreduce/share/hadoop/mapreduce/*
    STARTUP_MSG: build =
    file:///var/lib/jenkins/workspace/CDH4.1.2-Packaging-Hadoop/build/cdh4/hadoop/2.0.0-cdh4.1.2/source/hadoop-common-project/hadoop-common
    -r f0b53c81cbf56f5955e403b49fcd27afd5f082de; compiled by 'jenkins' on Thu
    Nov 1 17:06:55 PDT 2012
    ************************************************************/
    2012-12-02 10:32:22,214 INFO
    org.apache.hadoop.metrics2.impl.MetricsConfig:
    loaded properties from hadoop-metrics2.properties
    2012-12-02 10:32:23,240 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period
    at 10 second(s).
    2012-12-02 10:32:23,240 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
    started
    2012-12-02 10:32:25,269 INFO org.apache.hadoop.util.HostsFileReader:
    Refreshing hosts (include/exclude) list
    2012-12-02 10:32:25,270 INFO org.apache.hadoop.util.HostsFileReader: Adding
    127.0.0.1 to the list of hosts from
    /home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE/dfs_hosts_allow.txt
    2012-12-02 10:32:25,362 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
    dfs.block.invalidate.limit=1000
    2012-12-02 10:32:25,422 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    dfs.block.access.token.enable=false
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    defaultReplication = 1
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    maxReplication
    = 512
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    minReplication
    = 1
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    maxReplicationStreams = 2
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    shouldCheckForEnoughRacks = true
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    replicationRecheckInterval = 3000
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    encryptDataTransfer = false
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
    john (auth:SIMPLE)
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
    john
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
    true
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
    2012-12-02 10:32:25,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
    2012-12-02 10:32:27,741 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring
    more than 10 times
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.threshold-pct = 0.9990000128746033
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.min.datanodes = 0
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.extension = 30000
    2012-12-02 10:32:27,941 INFO
    org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /home/john/dfs_dir/namenode/data1/in_use.lock acquired by nodename
    2070@ubuntu1
    2012-12-02 10:32:28,574 INFO
    org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /home/john/dfs_dir/namenode/data2/in_use.lock acquired by nodename
    2070@ubuntu1
    2012-12-02 10:32:28,803 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering
    unfinalized segments in /home/john/dfs_dir/namenode/data1/current
    2012-12-02 10:32:28,803 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering
    unfinalized segments in /home/john/dfs_dir/namenode/data2/current
    2012-12-02 10:32:28,805 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    No edit log streams selected.
    2012-12-02 10:32:28,950 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Loading image file
    /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000 using
    no compression
    2012-12-02 10:32:28,950 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Number of files = 1
    2012-12-02 10:32:28,951 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Number of files under construction = 0
    2012-12-02 10:32:28,952 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Image file of size 117 loaded in 0 seconds.
    2012-12-02 10:32:28,952 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Loaded image for txid 0 from
    /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000
    2012-12-02 10:32:28,955 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 1
    2012-12-02 10:32:30,041 INFO
    org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with 0 entries
    0 lookups
    2012-12-02 10:32:30,041 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading
    FSImage in 2298 msecs
    2012-12-02 10:32:31,360 INFO org.apache.hadoop.ipc.Server: Starting Socket
    Reader #1 for port 8020
    2012-12-02 10:32:31,402 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
    FSNamesystemState MBean
    2012-12-02 10:32:31,482 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under
    construction: 0
    2012-12-02 10:32:31,482 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: initializing
    replication queues
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Total number of
    blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of
    invalid blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of
    under-replicated blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of
    over-replicated blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of blocks
    being written = 0
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange: STATE*
    Replication Queue initialization scan for invalid, over- and
    under-replicated blocks completed in 31 msec
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange: STATE*
    Leaving safe mode after 6 secs.
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange: STATE*
    Network topology has 0 racks and 0 datanodes
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange: STATE*
    UnderReplicatedBlocks has 0 blocks
    2012-12-02 10:32:31,674 INFO org.mortbay.log: Logging to
    org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
    org.mortbay.log.Slf4jLog
    2012-12-02 10:32:31,814 INFO org.apache.hadoop.http.HttpServer: Added global
    filter 'safety'
    (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer: Added filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
    context hdfs
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer: Added filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
    context logs
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer: Added filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
    context static
    2012-12-02 10:32:31,841 INFO org.apache.hadoop.http.HttpServer:
    dfs.webhdfs.enabled = true
    2012-12-02 10:32:31,842 INFO org.apache.hadoop.http.HttpServer: Added filter
    'SPNEGO' (class=org.apache.hadoop.hdfs.web.AuthFilter)
    2012-12-02 10:32:31,844 INFO org.apache.hadoop.http.HttpServer:
    addJerseyResourcePackage:
    packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
    pathSpec=/webhdfs/v1/*
    2012-12-02 10:32:31,862 INFO org.apache.hadoop.http.HttpServer: Jetty bound
    to port 50070
    2012-12-02 10:32:31,862 INFO org.mortbay.log: jetty-6.1.26.cloudera.2
    2012-12-02 10:32:32,261 WARN
    org.apache.hadoop.security.authentication.server.AuthenticationFilter:
    'signature.secret' configuration not set, using a random value as secret
    2012-12-02 10:32:32,453 INFO org.mortbay.log: Started
    selectchannelconnector@localhost.localdomain:50070
    2012-12-02 10:32:32,453 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at:
    localhost.localdomain:50070
    2012-12-02 10:32:32,453 INFO org.apache.hadoop.ipc.Server: IPC Server
    Responder: starting
    2012-12-02 10:32:32,454 INFO org.apache.hadoop.ipc.Server: IPC Server
    listener on 8020: starting
    2012-12-02 10:32:32,603 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode up at:
    ubuntu1/127.0.0.1:8020
    2012-12-02 10:32:32,603 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services
    required for active state
    2012-12-02 10:32:34,359 INFO org.apache.hadoop.fs.TrashPolicyDefault: The
    configured checkpoint interval is 0 minutes. Using an interval of 1440
    minutes that is used for deletion instead
    2012-12-02 10:32:35,780 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
    NameSystem.registerDatanode: node registration from
    DatanodeRegistration(127.0.0.1,
    storageID=DS-377461142-127.0.0.1-50010-1354473155123, infoPort=50075,
    ipcPort=50020, storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0) storage
    DS-377461142-127.0.0.1-50010-1354473155123
    2012-12-02 10:32:35,790 INFO org.apache.hadoop.net.NetworkTopology: Adding a
    new node: /default/127.0.0.1:50010
    2012-12-02 10:32:35,951 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: BLOCK*
    processReport: Received first block report from 127.0.0.1:50010 after
    becoming active. Its block contents are no longer considered stale.
    2012-12-02 10:32:35,951 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
    processReport: from DatanodeRegistration(127.0.0.1,
    storageID=DS-377461142-127.0.0.1-50010-1354473155123, infoPort=50075,
    ipcPort=50020, storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0), blocks:
    0, processing time: 1 msecs
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
    127.0.0.1
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs.
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 1
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
    transactions: 2
    Total time for transactions(ms): 1Number of transactions batched in Syncs: 0
    Number of syncs: 2 SyncTimes(ms): 166 189
    2012-12-02 10:33:31,595 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
    transactions: 2
    Total time for transactions(ms): 1Number of transactions batched in Syncs: 0
    Number of syncs: 3 SyncTimes(ms): 215 255
    2012-12-02 10:33:31,603 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
    file
    /home/john/dfs_dir/namenode/data1/current/edits_inprogress_0000000000000000001
    ->
    /home/john/dfs_dir/namenode/data1/current/edits_0000000000000000001-0000000000000000002
    2012-12-02 10:33:31,604 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
    file
    /home/john/dfs_dir/namenode/data2/current/edits_inprogress_0000000000000000001
    ->
    /home/john/dfs_dir/namenode/data2/current/edits_0000000000000000001-0000000000000000002
    2012-12-02 10:33:31,604 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 3
    2012-12-02 10:33:33,274 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Opening
    connection
    to
    http://ubuntu1:50090/getimage?getimage=1&txid=2&storageInfo=-40:829593698:0:cluster2
    2012-12-02 10:33:33,622 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Transfer took 0.34s
    at 0.00 KB/s
    2012-12-02 10:33:33,622 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Downloaded file
    fsimage.ckpt_0000000000000000002 size 117 bytes.
    2012-12-02 10:33:33,841 INFO
    org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Going to
    retain 2 images with txid >= 0



    --
    Harsh J


    --
    Thanks,
    *Shouvanik*
  • Harsh J at Dec 2, 2012 at 8:15 pm
    Hi Shouvanik,

    Those logs are all INFO level and there's no issue in it. Can you
    share the issue/message you get when you mean "I am not able to start
    hbase" instead?

    On Mon, Dec 3, 2012 at 1:35 AM, Shouvanik Haldar
    wrote:
    Hi Harsh,

    I am not able to start hbase although hbase service is up. I am getting
    weird zookeeper exception. can you please help

    [root@ip-10-40-222-77 ~]# tail -f
    /var/log/zookeeper/zookeeper-cmf-zookeeper1-SERVER-ip-10-40-222-77.ec2.internal.log
    2012-12-02 03:42:47,991 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
    KeeperException when processing sessionid:0x13b5ac2ad7a0002 type:create
    cxid:0x6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
    Path:/hbase/master Error:KeeperErrorCode = NodeExists for /hbase/master
    2012-12-02 03:42:48,001 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30002, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30003, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30001, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30002
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30003
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30001
    2012-12-02 03:42:48,182 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44800
    2012-12-02 03:42:48,194 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44800
    2012-12-02 03:42:48,196 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0003 with negotiated timeout 40000 for
    client /10.40.222.77:44800


    Regards,
    Shouvanik Haldar


    On Mon, Dec 3, 2012 at 1:24 AM, Harsh J wrote:

    I replied to this over your duplicate post at

    https://groups.google.com/a/cloudera.org/d/topic/cdh-user/ft02DH52BZ4/discussion
    On Mon, Dec 3, 2012 at 12:59 AM, John H wrote:

    Hi,

    I installed Cloudera Manager(4.1.1) and CDH4 (4.1.2) using tarball on
    Ubuntu 10.04.4 (VM using virtual box). I was able to setup Cloudera
    manager
    server and agent, setup roles (NN,DN and Secondary NN) in pseudo
    distributed
    mode. I deployed the configuration to hosts and formatted name node
    successfully.

    When I started the services from Cloudera manager, I got message saying
    all
    3 (NN,DN,SNN) were started successfully, but I cannot access anyone of
    them
    through web url.
    The status for all 3 is started with good health.


    I looked at the log file for namenode and it is stuck at this line



    "2012-12-02 10:33:33,841 INFO
    org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Going
    to
    retain 2 images with txid >= 0"

    This is the first start after format, so I am confused why this is
    happening... Any help would be really appreciated..


    Thanks
    JH





    FULL LOG FILE: ***************************



    ************************************************************/
    2012-12-02 10:32:19,931 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting NameNode
    STARTUP_MSG: host = ubuntu1/127.0.0.1
    STARTUP_MSG: args = []
    STARTUP_MSG: version = 2.0.0-cdh4.1.2
    STARTUP_MSG: classpath =

    /home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-log4j12-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-json-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jettison-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jets3t-0.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-digester-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-annotations-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/activation-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/kfs-0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-net-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-auth-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsch-0.1.42.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-api-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/paranamer-2.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-math-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/junit-4.8.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/avro-1.7.1.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/stax-api-1.0.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-tests.jar:/home/john/cloudera-manager/cm-4.1.1/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-daemon-1.0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-tests.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-sources.jar:/usr/lib/hadoop-yarn/share/hadoop/yarn/*:/usr/lib/hadoop-mapreduce/share/hadoop/mapreduce/*
    STARTUP_MSG: build =

    file:///var/lib/jenkins/workspace/CDH4.1.2-Packaging-Hadoop/build/cdh4/hadoop/2.0.0-cdh4.1.2/source/hadoop-common-project/hadoop-common
    -r f0b53c81cbf56f5955e403b49fcd27afd5f082de; compiled by 'jenkins' on
    Thu
    Nov 1 17:06:55 PDT 2012
    ************************************************************/
    2012-12-02 10:32:22,214 INFO
    org.apache.hadoop.metrics2.impl.MetricsConfig:
    loaded properties from hadoop-metrics2.properties
    2012-12-02 10:32:23,240 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
    period
    at 10 second(s).
    2012-12-02 10:32:23,240 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
    system
    started
    2012-12-02 10:32:25,269 INFO org.apache.hadoop.util.HostsFileReader:
    Refreshing hosts (include/exclude) list
    2012-12-02 10:32:25,270 INFO org.apache.hadoop.util.HostsFileReader:
    Adding
    127.0.0.1 to the list of hosts from

    /home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE/dfs_hosts_allow.txt
    2012-12-02 10:32:25,362 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
    dfs.block.invalidate.limit=1000
    2012-12-02 10:32:25,422 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    dfs.block.access.token.enable=false
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    defaultReplication = 1
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    maxReplication
    = 512
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    minReplication
    = 1
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    maxReplicationStreams = 2
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    shouldCheckForEnoughRacks = true
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    replicationRecheckInterval = 3000
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    encryptDataTransfer = false
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
    =
    john (auth:SIMPLE)
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
    =
    john
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled
    =
    true
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
    2012-12-02 10:32:25,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled:
    true
    2012-12-02 10:32:27,741 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
    occuring
    more than 10 times
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.threshold-pct = 0.9990000128746033
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.min.datanodes = 0
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.extension = 30000
    2012-12-02 10:32:27,941 INFO
    org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /home/john/dfs_dir/namenode/data1/in_use.lock acquired by
    nodename
    2070@ubuntu1
    2012-12-02 10:32:28,574 INFO
    org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /home/john/dfs_dir/namenode/data2/in_use.lock acquired by
    nodename
    2070@ubuntu1
    2012-12-02 10:32:28,803 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering
    unfinalized segments in /home/john/dfs_dir/namenode/data1/current
    2012-12-02 10:32:28,803 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering
    unfinalized segments in /home/john/dfs_dir/namenode/data2/current
    2012-12-02 10:32:28,805 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    No edit log streams selected.
    2012-12-02 10:32:28,950 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Loading image file
    /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000
    using
    no compression
    2012-12-02 10:32:28,950 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Number of files = 1
    2012-12-02 10:32:28,951 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Number of files under construction = 0
    2012-12-02 10:32:28,952 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Image file of size 117 loaded in 0 seconds.
    2012-12-02 10:32:28,952 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Loaded image for txid 0 from
    /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000
    2012-12-02 10:32:28,955 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment
    at 1
    2012-12-02 10:32:30,041 INFO
    org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with 0
    entries
    0 lookups
    2012-12-02 10:32:30,041 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading
    FSImage in 2298 msecs
    2012-12-02 10:32:31,360 INFO org.apache.hadoop.ipc.Server: Starting
    Socket
    Reader #1 for port 8020
    2012-12-02 10:32:31,402 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
    FSNamesystemState MBean
    2012-12-02 10:32:31,482 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks
    under
    construction: 0
    2012-12-02 10:32:31,482 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: initializing
    replication queues
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Total number
    of
    blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of
    invalid blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of
    under-replicated blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of
    over-replicated blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of
    blocks
    being written = 0
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange: STATE*
    Replication Queue initialization scan for invalid, over- and
    under-replicated blocks completed in 31 msec
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange: STATE*
    Leaving safe mode after 6 secs.
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange: STATE*
    Network topology has 0 racks and 0 datanodes
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange: STATE*
    UnderReplicatedBlocks has 0 blocks
    2012-12-02 10:32:31,674 INFO org.mortbay.log: Logging to
    org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
    org.mortbay.log.Slf4jLog
    2012-12-02 10:32:31,814 INFO org.apache.hadoop.http.HttpServer: Added
    global
    filter 'safety'
    (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer: Added
    filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context hdfs
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer: Added
    filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context logs
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer: Added
    filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context static
    2012-12-02 10:32:31,841 INFO org.apache.hadoop.http.HttpServer:
    dfs.webhdfs.enabled = true
    2012-12-02 10:32:31,842 INFO org.apache.hadoop.http.HttpServer: Added
    filter
    'SPNEGO' (class=org.apache.hadoop.hdfs.web.AuthFilter)
    2012-12-02 10:32:31,844 INFO org.apache.hadoop.http.HttpServer:
    addJerseyResourcePackage:

    packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
    pathSpec=/webhdfs/v1/*
    2012-12-02 10:32:31,862 INFO org.apache.hadoop.http.HttpServer: Jetty
    bound
    to port 50070
    2012-12-02 10:32:31,862 INFO org.mortbay.log: jetty-6.1.26.cloudera.2
    2012-12-02 10:32:32,261 WARN
    org.apache.hadoop.security.authentication.server.AuthenticationFilter:
    'signature.secret' configuration not set, using a random value as secret
    2012-12-02 10:32:32,453 INFO org.mortbay.log: Started
    selectchannelconnector@localhost.localdomain:50070
    2012-12-02 10:32:32,453 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at:
    localhost.localdomain:50070
    2012-12-02 10:32:32,453 INFO org.apache.hadoop.ipc.Server: IPC Server
    Responder: starting
    2012-12-02 10:32:32,454 INFO org.apache.hadoop.ipc.Server: IPC Server
    listener on 8020: starting
    2012-12-02 10:32:32,603 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode up at:
    ubuntu1/127.0.0.1:8020
    2012-12-02 10:32:32,603 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services
    required for active state
    2012-12-02 10:32:34,359 INFO org.apache.hadoop.fs.TrashPolicyDefault:
    The
    configured checkpoint interval is 0 minutes. Using an interval of 1440
    minutes that is used for deletion instead
    2012-12-02 10:32:35,780 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
    NameSystem.registerDatanode: node registration from
    DatanodeRegistration(127.0.0.1,
    storageID=DS-377461142-127.0.0.1-50010-1354473155123, infoPort=50075,
    ipcPort=50020, storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0)
    storage
    DS-377461142-127.0.0.1-50010-1354473155123
    2012-12-02 10:32:35,790 INFO org.apache.hadoop.net.NetworkTopology:
    Adding a
    new node: /default/127.0.0.1:50010
    2012-12-02 10:32:35,951 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: BLOCK*
    processReport: Received first block report from 127.0.0.1:50010 after
    becoming active. Its block contents are no longer considered stale.
    2012-12-02 10:32:35,951 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
    processReport: from DatanodeRegistration(127.0.0.1,
    storageID=DS-377461142-127.0.0.1-50010-1354473155123, infoPort=50075,
    ipcPort=50020, storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0),
    blocks:
    0, processing time: 1 msecs
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
    127.0.0.1
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs.
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 1
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
    transactions: 2
    Total time for transactions(ms): 1Number of transactions batched in
    Syncs: 0
    Number of syncs: 2 SyncTimes(ms): 166 189
    2012-12-02 10:33:31,595 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
    transactions: 2
    Total time for transactions(ms): 1Number of transactions batched in
    Syncs: 0
    Number of syncs: 3 SyncTimes(ms): 215 255
    2012-12-02 10:33:31,603 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing
    edits
    file

    /home/john/dfs_dir/namenode/data1/current/edits_inprogress_0000000000000000001
    ->

    /home/john/dfs_dir/namenode/data1/current/edits_0000000000000000001-0000000000000000002
    2012-12-02 10:33:31,604 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing
    edits
    file

    /home/john/dfs_dir/namenode/data2/current/edits_inprogress_0000000000000000001
    ->

    /home/john/dfs_dir/namenode/data2/current/edits_0000000000000000001-0000000000000000002
    2012-12-02 10:33:31,604 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment
    at 3
    2012-12-02 10:33:33,274 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Opening
    connection
    to

    http://ubuntu1:50090/getimage?getimage=1&txid=2&storageInfo=-40:829593698:0:cluster2
    2012-12-02 10:33:33,622 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Transfer took
    0.34s
    at 0.00 KB/s
    2012-12-02 10:33:33,622 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Downloaded file
    fsimage.ckpt_0000000000000000002 size 117 bytes.
    2012-12-02 10:33:33,841 INFO
    org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Going
    to
    retain 2 images with txid >= 0



    --
    Harsh J



    --
    Thanks,
    Shouvanik


    --
    Harsh J
  • Harsh J at Dec 2, 2012 at 8:34 pm
    Hi Shouvanik,

    I notice you use CDH3. You should really be using CDH4, as it provides
    a great deal of improvements over the CDH3 version that you'll be
    missing out on.

    In any case:

    If there exists a /etc/zookeeper/zoo.cfg on your client machine,
    please rename it to something else, such as zoo.cfg.unwanted, or
    remove it altogether, and retry.

    Alternatively, try to run your HBase shell command(s) with the direct
    script (instead of wrapper /usr/bin/hbase) as:

    /usr/lib/hbase/bin/hbase shell

    And see if it lets you connect and work normally this time.

    On Mon, Dec 3, 2012 at 1:50 AM, Shouvanik Haldar
    wrote:
    Yes Harsh,

    [root@ip-10-40-222-77 ~]# hbase shell
    HBase Shell; enter 'help<RETURN>' for list of supported commands.
    Type "exit<RETURN>" to leave the HBase Shell
    Version 0.90.4-cdh3u3, r, Thu Jan 26 10:13:36 PST 2012

    hbase(main):001:0> describe 't1'

    ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able
    to connect to ZooKeeper but the connection closes immediately. This could be
    a sign that the server has too many connections (30 is the default).
    Consider inspecting your ZK server logs for that error and then make sure
    you are reusing HBaseConfiguration as often as you can. See HTable's javadoc
    for more information.

    Here is some help for this command:
    Describe the named table. For example:
    hbase> describe 't1'


    hbase(main):002:0>


    Secondly,

    When I see the zookeeper logs, it says

    [root@ip-10-40-222-77 ~]# cat
    /var/log/zookeeper/zookeeper-cmf-zookeeper1-SERVER-ip-10-40-222-77.ec2.internal.log
    2012-12-01 23:30:35,726 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading configuration
    from: /var/run/cloudera-scm-agent/process/1-zookeeper-server/zoo.cfg
    2012-12-01 23:30:35,731 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-01 23:30:35,811 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading configuration
    from: /var/run/cloudera-scm-agent/process/1-zookeeper-server/zoo.cfg
    2012-12-01 23:30:35,812 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-01 23:30:35,842 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on 01/26/2012
    20:08 GMT
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/1-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/1-zookeeper-server
    2012-12-01 23:30:35,851 INFO org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-01 23:30:35,851 INFO org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-01 23:30:35,851 INFO org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-01 23:30:35,891 INFO org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-01 23:30:35,907 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.0
    2012-12-01 23:30:35,941 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 28
    2012-12-01 23:38:21,260 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading configuration
    from: /var/run/cloudera-scm-agent/process/9-zookeeper-server/zoo.cfg
    2012-12-01 23:38:21,265 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-01 23:38:21,339 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading configuration
    from: /var/run/cloudera-scm-agent/process/9-zookeeper-server/zoo.cfg
    2012-12-01 23:38:21,340 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-01 23:38:21,357 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on 01/26/2012
    20:08 GMT
    2012-12-01 23:38:21,357 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-01 23:38:21,357 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-01 23:38:21,357 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-01 23:38:21,357 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-01 23:38:21,357 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/9-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-01 23:38:21,357 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-01 23:38:21,357 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-01 23:38:21,357 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-01 23:38:21,358 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-01 23:38:21,358 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-01 23:38:21,358 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-01 23:38:21,358 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-01 23:38:21,358 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-01 23:38:21,358 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/9-zookeeper-server
    2012-12-01 23:38:21,366 INFO org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-01 23:38:21,366 INFO org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-01 23:38:21,366 INFO org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-01 23:38:21,399 INFO org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-01 23:38:21,415 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.28
    2012-12-01 23:38:21,430 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 28
    2012-12-01 23:43:49,121 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:45741
    2012-12-01 23:43:49,128 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:45741
    2012-12-01 23:43:49,130 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.29
    2012-12-01 23:43:49,172 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0000 with negotiated timeout 40000 for
    client /10.83.35.173:45741
    2012-12-01 23:43:52,894 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:59824
    2012-12-01 23:43:52,897 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:59824
    2012-12-01 23:43:52,899 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0001 with negotiated timeout 40000 for
    client /10.40.222.77:59824
    2012-12-01 23:43:52,993 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:59825
    2012-12-01 23:43:52,996 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:59825
    2012-12-01 23:43:52,998 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0002 with negotiated timeout 40000 for
    client /10.40.222.77:59825
    2012-12-02 02:48:13,910 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading configuration
    from: /var/run/cloudera-scm-agent/process/34-zookeeper-server/zoo.cfg
    2012-12-02 02:48:13,914 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-02 02:48:14,002 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading configuration
    from: /var/run/cloudera-scm-agent/process/34-zookeeper-server/zoo.cfg
    2012-12-02 02:48:14,003 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-02 02:48:14,030 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on 01/26/2012
    20:08 GMT
    2012-12-02 02:48:14,030 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-02 02:48:14,030 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/34-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/34-zookeeper-server
    2012-12-02 02:48:14,039 INFO org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-02 02:48:14,039 INFO org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-02 02:48:14,040 INFO org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-02 02:48:14,072 INFO org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-02 02:48:14,088 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.28
    2012-12-02 02:48:14,109 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 2c
    2012-12-02 02:48:15,025 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:39458
    2012-12-02 02:48:15,029 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0002 at /10.40.222.77:39458
    2012-12-02 02:48:15,035 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0002 with negotiated timeout 40000 for
    client /10.40.222.77:39458
    2012-12-02 02:48:15,060 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:56157
    2012-12-02 02:48:15,062 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0000 at /10.83.35.173:56157
    2012-12-02 02:48:15,063 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0000 with negotiated timeout 40000 for
    client /10.83.35.173:56157
    2012-12-02 02:48:15,095 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:39459
    2012-12-02 02:48:15,095 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0001 at /10.40.222.77:39459
    2012-12-02 02:48:15,096 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0001 with negotiated timeout 40000 for
    client /10.40.222.77:39459
    2012-12-02 03:23:48,488 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.2d
    2012-12-02 03:23:48,560 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0001
    2012-12-02 03:23:48,570 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:39459 which had sessionid
    0x13b59e867fc0001
    2012-12-02 03:25:46,004 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54443
    2012-12-02 03:25:46,007 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54443
    2012-12-02 03:25:46,009 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30000 with negotiated timeout 40000 for
    client /10.40.222.77:54443
    2012-12-02 03:25:46,348 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30000, likely client has closed socket
    2012-12-02 03:25:46,349 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:54443 which had sessionid
    0x13b5a963ea30000
    2012-12-02 03:26:15,044 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b59e867fc0000, likely client has closed socket
    2012-12-02 03:26:15,045 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.83.35.173:56157 which had sessionid
    0x13b59e867fc0000
    2012-12-02 03:26:17,867 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b59e867fc0002, likely client has closed socket
    2012-12-02 03:26:17,867 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:39458 which had sessionid
    0x13b59e867fc0002
    2012-12-02 03:26:28,001 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30000, timeout of 40000ms exceeded
    2012-12-02 03:26:28,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30000
    2012-12-02 03:26:42,001 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b59e867fc0000, timeout of 40000ms exceeded
    2012-12-02 03:26:42,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0000
    2012-12-02 03:26:47,391 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54461
    2012-12-02 03:26:47,394 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54461
    2012-12-02 03:26:47,396 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30001 with negotiated timeout 40000 for
    client /10.40.222.77:54461
    2012-12-02 03:26:47,448 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54462
    2012-12-02 03:26:47,460 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54462
    2012-12-02 03:26:47,463 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30002 with negotiated timeout 40000 for
    client /10.40.222.77:54462
    2012-12-02 03:26:56,000 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b59e867fc0002, timeout of 40000ms exceeded
    2012-12-02 03:26:56,000 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0002
    2012-12-02 03:27:00,606 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:56794
    2012-12-02 03:27:00,614 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:56794
    2012-12-02 03:27:00,617 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30003 with negotiated timeout 40000 for
    client /10.83.35.173:56794
    2012-12-02 03:36:45,885 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading configuration
    from: /var/run/cloudera-scm-agent/process/43-zookeeper-server/zoo.cfg
    2012-12-02 03:36:45,889 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-02 03:36:45,962 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading configuration
    from: /var/run/cloudera-scm-agent/process/43-zookeeper-server/zoo.cfg
    2012-12-02 03:36:45,962 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on 01/26/2012
    20:08 GMT
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/43-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/43-zookeeper-server
    2012-12-02 03:36:45,985 INFO org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-02 03:36:45,985 INFO org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-02 03:36:45,985 INFO org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-02 03:36:46,024 INFO org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-02 03:36:46,042 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.2c
    2012-12-02 03:36:46,063 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 36
    2012-12-02 03:36:46,531 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:37153
    2012-12-02 03:36:46,535 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30003 at /10.83.35.173:37153
    2012-12-02 03:36:46,539 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30003 with negotiated timeout 40000 for
    client /10.83.35.173:37153
    2012-12-02 03:36:46,883 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:47487
    2012-12-02 03:36:46,884 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30002 at /10.40.222.77:47487
    2012-12-02 03:36:46,884 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30002 with negotiated timeout 40000 for
    client /10.40.222.77:47487
    2012-12-02 03:36:47,310 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:47488
    2012-12-02 03:36:47,311 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30001 at /10.40.222.77:47488
    2012-12-02 03:36:47,312 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30001 with negotiated timeout 40000 for
    client /10.40.222.77:47488
    2012-12-02 03:41:46,616 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44776
    2012-12-02 03:41:46,619 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44776
    2012-12-02 03:41:46,620 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.37
    2012-12-02 03:41:46,674 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0000 with negotiated timeout 40000 for
    client /10.40.222.77:44776
    2012-12-02 03:41:47,494 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5ac2ad7a0000, likely client has closed socket
    2012-12-02 03:41:47,495 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:44776 which had sessionid
    0x13b5ac2ad7a0000
    2012-12-02 03:42:16,001 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30003, likely client has closed socket
    2012-12-02 03:42:16,002 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.83.35.173:37153 which had sessionid
    0x13b5a963ea30003
    2012-12-02 03:42:17,916 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30002, likely client has closed socket
    2012-12-02 03:42:17,917 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:47487 which had sessionid
    0x13b5a963ea30002
    2012-12-02 03:42:19,468 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30001, likely client has closed socket
    2012-12-02 03:42:19,469 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:47488 which had sessionid
    0x13b5a963ea30001
    2012-12-02 03:42:28,000 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5ac2ad7a0000, timeout of 40000ms exceeded
    2012-12-02 03:42:28,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5ac2ad7a0000
    2012-12-02 03:42:46,214 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:52486
    2012-12-02 03:42:46,217 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:52486
    2012-12-02 03:42:46,220 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0001 with negotiated timeout 40000 for
    client /10.83.35.173:52486
    2012-12-02 03:42:47,857 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44799
    2012-12-02 03:42:47,864 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44799
    2012-12-02 03:42:47,867 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0002 with negotiated timeout 40000 for
    client /10.40.222.77:44799

    2012-12-02 03:42:47,991 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
    KeeperException when processing sessionid:0x13b5ac2ad7a0002 type:create
    cxid:0x6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
    Path:/hbase/master Error:KeeperErrorCode = NodeExists for /hbase/master
    2012-12-02 03:42:48,001 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30002, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30003, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30001, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30002
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30003
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30001
    2012-12-02 03:42:48,182 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44800
    2012-12-02 03:42:48,194 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44800
    2012-12-02 03:42:48,196 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0003 with negotiated timeout 40000 for
    client /10.40.222.77:44800


    Regards,
    Shouvanik


    On Mon, Dec 3, 2012 at 1:44 AM, Harsh J wrote:

    Hi Shouvanik,

    Those logs are all INFO level and there's no issue in it. Can you
    share the issue/message you get when you mean "I am not able to start
    hbase" instead?

    On Mon, Dec 3, 2012 at 1:35 AM, Shouvanik Haldar
    wrote:
    Hi Harsh,

    I am not able to start hbase although hbase service is up. I am getting
    weird zookeeper exception. can you please help

    [root@ip-10-40-222-77 ~]# tail -f

    /var/log/zookeeper/zookeeper-cmf-zookeeper1-SERVER-ip-10-40-222-77.ec2.internal.log
    2012-12-02 03:42:47,991 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
    KeeperException when processing sessionid:0x13b5ac2ad7a0002 type:create
    cxid:0x6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
    Path:/hbase/master Error:KeeperErrorCode = NodeExists for /hbase/master
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30002, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30003, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30001, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30002
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30003
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30001
    2012-12-02 03:42:48,182 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44800
    2012-12-02 03:42:48,194 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44800
    2012-12-02 03:42:48,196 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0003 with negotiated timeout 40000 for
    client /10.40.222.77:44800


    Regards,
    Shouvanik Haldar


    On Mon, Dec 3, 2012 at 1:24 AM, Harsh J wrote:

    I replied to this over your duplicate post at


    https://groups.google.com/a/cloudera.org/d/topic/cdh-user/ft02DH52BZ4/discussion
    On Mon, Dec 3, 2012 at 12:59 AM, John H wrote:

    Hi,

    I installed Cloudera Manager(4.1.1) and CDH4 (4.1.2) using tarball
    on
    Ubuntu 10.04.4 (VM using virtual box). I was able to setup Cloudera
    manager
    server and agent, setup roles (NN,DN and Secondary NN) in pseudo
    distributed
    mode. I deployed the configuration to hosts and formatted name node
    successfully.

    When I started the services from Cloudera manager, I got message
    saying
    all
    3 (NN,DN,SNN) were started successfully, but I cannot access anyone
    of
    them
    through web url.
    The status for all 3 is started with good health.


    I looked at the log file for namenode and it is stuck at this line



    "2012-12-02 10:33:33,841 INFO
    org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager:
    Going
    to
    retain 2 images with txid >= 0"

    This is the first start after format, so I am confused why this is
    happening... Any help would be really appreciated..


    Thanks
    JH





    FULL LOG FILE: ***************************



    ************************************************************/
    2012-12-02 10:32:19,931 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting NameNode
    STARTUP_MSG: host = ubuntu1/127.0.0.1
    STARTUP_MSG: args = []
    STARTUP_MSG: version = 2.0.0-cdh4.1.2
    STARTUP_MSG: classpath =


    /home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-log4j12-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-json-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jettison-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jets3t-0.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-digester-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-annotations-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/activation-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/kfs-0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-net-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-auth-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsch-0.1.42.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-api-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/paranamer-2.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-math-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/junit-4.8.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/avro-1.7.1.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/stax-api-1.0.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-tests.jar:/home/john/cloudera-manager/cm-4.1.1/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-daemon-1.0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-tests.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-sources.jar:/usr/lib/hadoop-yarn/share/hadoop/yarn/*:/usr/lib/hadoop-mapreduce/share/hadoop/mapreduce/*
    STARTUP_MSG: build =


    file:///var/lib/jenkins/workspace/CDH4.1.2-Packaging-Hadoop/build/cdh4/hadoop/2.0.0-cdh4.1.2/source/hadoop-common-project/hadoop-common
    -r f0b53c81cbf56f5955e403b49fcd27afd5f082de; compiled by 'jenkins' on
    Thu
    Nov 1 17:06:55 PDT 2012
    ************************************************************/
    2012-12-02 10:32:22,214 INFO
    org.apache.hadoop.metrics2.impl.MetricsConfig:
    loaded properties from hadoop-metrics2.properties
    2012-12-02 10:32:23,240 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
    period
    at 10 second(s).
    2012-12-02 10:32:23,240 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
    system
    started
    2012-12-02 10:32:25,269 INFO org.apache.hadoop.util.HostsFileReader:
    Refreshing hosts (include/exclude) list
    2012-12-02 10:32:25,270 INFO org.apache.hadoop.util.HostsFileReader:
    Adding
    127.0.0.1 to the list of hosts from


    /home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE/dfs_hosts_allow.txt
    2012-12-02 10:32:25,362 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
    dfs.block.invalidate.limit=1000
    2012-12-02 10:32:25,422 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    dfs.block.access.token.enable=false
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    defaultReplication = 1
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    maxReplication
    = 512
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    minReplication
    = 1
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    maxReplicationStreams = 2
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    shouldCheckForEnoughRacks = true
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    replicationRecheckInterval = 3000
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    encryptDataTransfer = false
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
    =
    john (auth:SIMPLE)
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
    =
    john
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    isPermissionEnabled
    =
    true
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled:
    false
    2012-12-02 10:32:25,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled:
    true
    2012-12-02 10:32:27,741 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
    occuring
    more than 10 times
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.threshold-pct = 0.9990000128746033
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.min.datanodes = 0
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.extension = 30000
    2012-12-02 10:32:27,941 INFO
    org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /home/john/dfs_dir/namenode/data1/in_use.lock acquired by
    nodename
    2070@ubuntu1
    2012-12-02 10:32:28,574 INFO
    org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /home/john/dfs_dir/namenode/data2/in_use.lock acquired by
    nodename
    2070@ubuntu1
    2012-12-02 10:32:28,803 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering
    unfinalized segments in /home/john/dfs_dir/namenode/data1/current
    2012-12-02 10:32:28,803 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering
    unfinalized segments in /home/john/dfs_dir/namenode/data2/current
    2012-12-02 10:32:28,805 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    No edit log streams selected.
    2012-12-02 10:32:28,950 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Loading image file
    /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000
    using
    no compression
    2012-12-02 10:32:28,950 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Number of files = 1
    2012-12-02 10:32:28,951 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Number of files under construction = 0
    2012-12-02 10:32:28,952 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Image file of size 117 loaded in 0 seconds.
    2012-12-02 10:32:28,952 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Loaded image for txid 0 from
    /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000
    2012-12-02 10:32:28,955 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
    segment
    at 1
    2012-12-02 10:32:30,041 INFO
    org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with 0
    entries
    0 lookups
    2012-12-02 10:32:30,041 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading
    FSImage in 2298 msecs
    2012-12-02 10:32:31,360 INFO org.apache.hadoop.ipc.Server: Starting
    Socket
    Reader #1 for port 8020
    2012-12-02 10:32:31,402 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
    FSNamesystemState MBean
    2012-12-02 10:32:31,482 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks
    under
    construction: 0
    2012-12-02 10:32:31,482 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: initializing
    replication queues
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Total
    number
    of
    blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of
    invalid blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of
    under-replicated blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of
    over-replicated blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of
    blocks
    being written = 0
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    Replication Queue initialization scan for invalid, over- and
    under-replicated blocks completed in 31 msec
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    Leaving safe mode after 6 secs.
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    Network topology has 0 racks and 0 datanodes
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    UnderReplicatedBlocks has 0 blocks
    2012-12-02 10:32:31,674 INFO org.mortbay.log: Logging to
    org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
    org.mortbay.log.Slf4jLog
    2012-12-02 10:32:31,814 INFO org.apache.hadoop.http.HttpServer: Added
    global
    filter 'safety'
    (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer: Added
    filter
    static_user_filter

    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context hdfs
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer: Added
    filter
    static_user_filter

    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context logs
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer: Added
    filter
    static_user_filter

    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context static
    2012-12-02 10:32:31,841 INFO org.apache.hadoop.http.HttpServer:
    dfs.webhdfs.enabled = true
    2012-12-02 10:32:31,842 INFO org.apache.hadoop.http.HttpServer: Added
    filter
    'SPNEGO' (class=org.apache.hadoop.hdfs.web.AuthFilter)
    2012-12-02 10:32:31,844 INFO org.apache.hadoop.http.HttpServer:
    addJerseyResourcePackage:


    packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
    pathSpec=/webhdfs/v1/*
    2012-12-02 10:32:31,862 INFO org.apache.hadoop.http.HttpServer: Jetty
    bound
    to port 50070
    2012-12-02 10:32:31,862 INFO org.mortbay.log: jetty-6.1.26.cloudera.2
    2012-12-02 10:32:32,261 WARN

    org.apache.hadoop.security.authentication.server.AuthenticationFilter:
    'signature.secret' configuration not set, using a random value as
    secret
    2012-12-02 10:32:32,453 INFO org.mortbay.log: Started
    selectchannelconnector@localhost.localdomain:50070
    2012-12-02 10:32:32,453 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at:
    localhost.localdomain:50070
    2012-12-02 10:32:32,453 INFO org.apache.hadoop.ipc.Server: IPC Server
    Responder: starting
    2012-12-02 10:32:32,454 INFO org.apache.hadoop.ipc.Server: IPC Server
    listener on 8020: starting
    2012-12-02 10:32:32,603 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode up at:
    ubuntu1/127.0.0.1:8020
    2012-12-02 10:32:32,603 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting
    services
    required for active state
    2012-12-02 10:32:34,359 INFO org.apache.hadoop.fs.TrashPolicyDefault:
    The
    configured checkpoint interval is 0 minutes. Using an interval of
    1440
    minutes that is used for deletion instead
    2012-12-02 10:32:35,780 INFO org.apache.hadoop.hdfs.StateChange:
    BLOCK*
    NameSystem.registerDatanode: node registration from
    DatanodeRegistration(127.0.0.1,
    storageID=DS-377461142-127.0.0.1-50010-1354473155123, infoPort=50075,
    ipcPort=50020, storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0)
    storage
    DS-377461142-127.0.0.1-50010-1354473155123
    2012-12-02 10:32:35,790 INFO org.apache.hadoop.net.NetworkTopology:
    Adding a
    new node: /default/127.0.0.1:50010
    2012-12-02 10:32:35,951 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: BLOCK*
    processReport: Received first block report from 127.0.0.1:50010 after
    becoming active. Its block contents are no longer considered stale.
    2012-12-02 10:32:35,951 INFO org.apache.hadoop.hdfs.StateChange:
    BLOCK*
    processReport: from DatanodeRegistration(127.0.0.1,
    storageID=DS-377461142-127.0.0.1-50010-1354473155123, infoPort=50075,
    ipcPort=50020, storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0),
    blocks:
    0, processing time: 1 msecs
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
    from
    127.0.0.1
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs.
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment
    1
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
    transactions: 2
    Total time for transactions(ms): 1Number of transactions batched in
    Syncs: 0
    Number of syncs: 2 SyncTimes(ms): 166 189
    2012-12-02 10:33:31,595 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
    transactions: 2
    Total time for transactions(ms): 1Number of transactions batched in
    Syncs: 0
    Number of syncs: 3 SyncTimes(ms): 215 255
    2012-12-02 10:33:31,603 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing
    edits
    file


    /home/john/dfs_dir/namenode/data1/current/edits_inprogress_0000000000000000001
    ->


    /home/john/dfs_dir/namenode/data1/current/edits_0000000000000000001-0000000000000000002
    2012-12-02 10:33:31,604 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing
    edits
    file


    /home/john/dfs_dir/namenode/data2/current/edits_inprogress_0000000000000000001
    ->


    /home/john/dfs_dir/namenode/data2/current/edits_0000000000000000001-0000000000000000002
    2012-12-02 10:33:31,604 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
    segment
    at 3
    2012-12-02 10:33:33,274 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Opening
    connection
    to


    http://ubuntu1:50090/getimage?getimage=1&txid=2&storageInfo=-40:829593698:0:cluster2
    2012-12-02 10:33:33,622 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Transfer took
    0.34s
    at 0.00 KB/s
    2012-12-02 10:33:33,622 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Downloaded
    file
    fsimage.ckpt_0000000000000000002 size 117 bytes.
    2012-12-02 10:33:33,841 INFO
    org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager:
    Going
    to
    retain 2 images with txid >= 0



    --
    Harsh J



    --
    Thanks,
    Shouvanik


    --
    Harsh J



    --
    Thanks,
    Shouvanik


    --
    Harsh J
  • Shouvanik Haldar at Dec 3, 2012 at 5:29 am
    Hi Harsh,

    I performed the following steps as advised by you

    1. If there exists a /etc/zookeeper/zoo.cfg on your client machine,
    please rename it to something else, such as zoo.cfg.unwanted, or
    remove it altogether, and retry.

    2. Alternatively, try to run your HBase shell command(s) with the direct
    script (instead of wrapper /usr/bin/hbase) as:

    /usr/lib/hbase/bin/hbase shell
    .


    But, still I am getting the below error

    *hbase(main):001:0> describe 't1'

    ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able
    to connect to ZooKeeper but the connection closes immediately. This could
    be a sign that the server has too many connections (30 is the default).
    Consider inspecting your ZK server logs for that error and then make sure
    you are reusing HBaseConfiguration as often as you can. See HTable's
    javadoc for more information.

    Here is some help for this command:
    Describe the named table. For example:
    hbase> describe 't1'*


    Please help!
    Regards,
    Shouvanik
    On Mon, Dec 3, 2012 at 2:04 AM, Harsh J wrote:

    Hi Shouvanik,

    I notice you use CDH3. You should really be using CDH4, as it provides
    a great deal of improvements over the CDH3 version that you'll be
    missing out on.

    In any case:

    If there exists a /etc/zookeeper/zoo.cfg on your client machine,
    please rename it to something else, such as zoo.cfg.unwanted, or
    remove it altogether, and retry.

    Alternatively, try to run your HBase shell command(s) with the direct
    script (instead of wrapper /usr/bin/hbase) as:

    /usr/lib/hbase/bin/hbase shell

    And see if it lets you connect and work normally this time.

    On Mon, Dec 3, 2012 at 1:50 AM, Shouvanik Haldar
    wrote:
    Yes Harsh,

    [root@ip-10-40-222-77 ~]# hbase shell
    HBase Shell; enter 'help<RETURN>' for list of supported commands.
    Type "exit<RETURN>" to leave the HBase Shell
    Version 0.90.4-cdh3u3, r, Thu Jan 26 10:13:36 PST 2012

    hbase(main):001:0> describe 't1'

    ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able
    to connect to ZooKeeper but the connection closes immediately. This could be
    a sign that the server has too many connections (30 is the default).
    Consider inspecting your ZK server logs for that error and then make sure
    you are reusing HBaseConfiguration as often as you can. See HTable's javadoc
    for more information.

    Here is some help for this command:
    Describe the named table. For example:
    hbase> describe 't1'


    hbase(main):002:0>


    Secondly,

    When I see the zookeeper logs, it says

    [root@ip-10-40-222-77 ~]# cat
    /var/log/zookeeper/zookeeper-cmf-zookeeper1-SERVER-ip-10-40-222-77.ec2.internal.log
    2012-12-01 23:30:35,726 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/1-zookeeper-server/zoo.cfg
    2012-12-01 23:30:35,731 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-01 23:30:35,811 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/1-zookeeper-server/zoo.cfg
    2012-12-01 23:30:35,812 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-01 23:30:35,842 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on 01/26/2012
    20:08 GMT
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/1-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/1-zookeeper-server
    2012-12-01 23:30:35,851 INFO org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-01 23:30:35,851 INFO org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-01 23:30:35,851 INFO org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-01 23:30:35,891 INFO org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-01 23:30:35,907 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.0
    2012-12-01 23:30:35,941 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 28
    2012-12-01 23:38:21,260 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/9-zookeeper-server/zoo.cfg
    2012-12-01 23:38:21,265 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-01 23:38:21,339 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/9-zookeeper-server/zoo.cfg
    2012-12-01 23:38:21,340 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-01 23:38:21,357 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on 01/26/2012
    20:08 GMT
    2012-12-01 23:38:21,357 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-01 23:38:21,357 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-01 23:38:21,357 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-01 23:38:21,357 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-01 23:38:21,357 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/9-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-01 23:38:21,357 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-01 23:38:21,357 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-01 23:38:21,357 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-01 23:38:21,358 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-01 23:38:21,358 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-01 23:38:21,358 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-01 23:38:21,358 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-01 23:38:21,358 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-01 23:38:21,358 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/9-zookeeper-server
    2012-12-01 23:38:21,366 INFO org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-01 23:38:21,366 INFO org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-01 23:38:21,366 INFO org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-01 23:38:21,399 INFO org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-01 23:38:21,415 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.28
    2012-12-01 23:38:21,430 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 28
    2012-12-01 23:43:49,121 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:45741
    2012-12-01 23:43:49,128 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:45741
    2012-12-01 23:43:49,130 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.29
    2012-12-01 23:43:49,172 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0000 with negotiated timeout 40000 for
    client /10.83.35.173:45741
    2012-12-01 23:43:52,894 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:59824
    2012-12-01 23:43:52,897 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:59824
    2012-12-01 23:43:52,899 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0001 with negotiated timeout 40000 for
    client /10.40.222.77:59824
    2012-12-01 23:43:52,993 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:59825
    2012-12-01 23:43:52,996 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:59825
    2012-12-01 23:43:52,998 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0002 with negotiated timeout 40000 for
    client /10.40.222.77:59825
    2012-12-02 02:48:13,910 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/34-zookeeper-server/zoo.cfg
    2012-12-02 02:48:13,914 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-02 02:48:14,002 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/34-zookeeper-server/zoo.cfg
    2012-12-02 02:48:14,003 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-02 02:48:14,030 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on 01/26/2012
    20:08 GMT
    2012-12-02 02:48:14,030 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-02 02:48:14,030 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/34-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/34-zookeeper-server
    2012-12-02 02:48:14,039 INFO org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-02 02:48:14,039 INFO org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-02 02:48:14,040 INFO org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-02 02:48:14,072 INFO org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-02 02:48:14,088 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.28
    2012-12-02 02:48:14,109 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 2c
    2012-12-02 02:48:15,025 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:39458
    2012-12-02 02:48:15,029 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0002 at /
    10.40.222.77:39458
    2012-12-02 02:48:15,035 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0002 with negotiated timeout 40000 for
    client /10.40.222.77:39458
    2012-12-02 02:48:15,060 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:56157
    2012-12-02 02:48:15,062 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0000 at /
    10.83.35.173:56157
    2012-12-02 02:48:15,063 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0000 with negotiated timeout 40000 for
    client /10.83.35.173:56157
    2012-12-02 02:48:15,095 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:39459
    2012-12-02 02:48:15,095 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0001 at /
    10.40.222.77:39459
    2012-12-02 02:48:15,096 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0001 with negotiated timeout 40000 for
    client /10.40.222.77:39459
    2012-12-02 03:23:48,488 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.2d
    2012-12-02 03:23:48,560 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0001
    2012-12-02 03:23:48,570 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:39459 which had sessionid
    0x13b59e867fc0001
    2012-12-02 03:25:46,004 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54443
    2012-12-02 03:25:46,007 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54443
    2012-12-02 03:25:46,009 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30000 with negotiated timeout 40000 for
    client /10.40.222.77:54443
    2012-12-02 03:25:46,348 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30000, likely client has closed socket
    2012-12-02 03:25:46,349 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:54443 which had sessionid
    0x13b5a963ea30000
    2012-12-02 03:26:15,044 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b59e867fc0000, likely client has closed socket
    2012-12-02 03:26:15,045 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.83.35.173:56157 which had sessionid
    0x13b59e867fc0000
    2012-12-02 03:26:17,867 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b59e867fc0002, likely client has closed socket
    2012-12-02 03:26:17,867 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:39458 which had sessionid
    0x13b59e867fc0002
    2012-12-02 03:26:28,001 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30000, timeout of 40000ms exceeded
    2012-12-02 03:26:28,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30000
    2012-12-02 03:26:42,001 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b59e867fc0000, timeout of 40000ms exceeded
    2012-12-02 03:26:42,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0000
    2012-12-02 03:26:47,391 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54461
    2012-12-02 03:26:47,394 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54461
    2012-12-02 03:26:47,396 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30001 with negotiated timeout 40000 for
    client /10.40.222.77:54461
    2012-12-02 03:26:47,448 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54462
    2012-12-02 03:26:47,460 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54462
    2012-12-02 03:26:47,463 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30002 with negotiated timeout 40000 for
    client /10.40.222.77:54462
    2012-12-02 03:26:56,000 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b59e867fc0002, timeout of 40000ms exceeded
    2012-12-02 03:26:56,000 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0002
    2012-12-02 03:27:00,606 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:56794
    2012-12-02 03:27:00,614 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:56794
    2012-12-02 03:27:00,617 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30003 with negotiated timeout 40000 for
    client /10.83.35.173:56794
    2012-12-02 03:36:45,885 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/43-zookeeper-server/zoo.cfg
    2012-12-02 03:36:45,889 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-02 03:36:45,962 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/43-zookeeper-server/zoo.cfg
    2012-12-02 03:36:45,962 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on 01/26/2012
    20:08 GMT
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/43-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/43-zookeeper-server
    2012-12-02 03:36:45,985 INFO org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-02 03:36:45,985 INFO org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-02 03:36:45,985 INFO org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-02 03:36:46,024 INFO org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-02 03:36:46,042 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.2c
    2012-12-02 03:36:46,063 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 36
    2012-12-02 03:36:46,531 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:37153
    2012-12-02 03:36:46,535 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30003 at /
    10.83.35.173:37153
    2012-12-02 03:36:46,539 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30003 with negotiated timeout 40000 for
    client /10.83.35.173:37153
    2012-12-02 03:36:46,883 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:47487
    2012-12-02 03:36:46,884 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30002 at /
    10.40.222.77:47487
    2012-12-02 03:36:46,884 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30002 with negotiated timeout 40000 for
    client /10.40.222.77:47487
    2012-12-02 03:36:47,310 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:47488
    2012-12-02 03:36:47,311 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30001 at /
    10.40.222.77:47488
    2012-12-02 03:36:47,312 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30001 with negotiated timeout 40000 for
    client /10.40.222.77:47488
    2012-12-02 03:41:46,616 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44776
    2012-12-02 03:41:46,619 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44776
    2012-12-02 03:41:46,620 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.37
    2012-12-02 03:41:46,674 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0000 with negotiated timeout 40000 for
    client /10.40.222.77:44776
    2012-12-02 03:41:47,494 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5ac2ad7a0000, likely client has closed socket
    2012-12-02 03:41:47,495 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:44776 which had sessionid
    0x13b5ac2ad7a0000
    2012-12-02 03:42:16,001 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30003, likely client has closed socket
    2012-12-02 03:42:16,002 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.83.35.173:37153 which had sessionid
    0x13b5a963ea30003
    2012-12-02 03:42:17,916 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30002, likely client has closed socket
    2012-12-02 03:42:17,917 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:47487 which had sessionid
    0x13b5a963ea30002
    2012-12-02 03:42:19,468 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30001, likely client has closed socket
    2012-12-02 03:42:19,469 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:47488 which had sessionid
    0x13b5a963ea30001
    2012-12-02 03:42:28,000 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5ac2ad7a0000, timeout of 40000ms exceeded
    2012-12-02 03:42:28,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5ac2ad7a0000
    2012-12-02 03:42:46,214 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:52486
    2012-12-02 03:42:46,217 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:52486
    2012-12-02 03:42:46,220 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0001 with negotiated timeout 40000 for
    client /10.83.35.173:52486
    2012-12-02 03:42:47,857 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44799
    2012-12-02 03:42:47,864 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44799
    2012-12-02 03:42:47,867 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0002 with negotiated timeout 40000 for
    client /10.40.222.77:44799

    2012-12-02 03:42:47,991 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
    KeeperException when processing sessionid:0x13b5ac2ad7a0002 type:create
    cxid:0x6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
    Path:/hbase/master Error:KeeperErrorCode = NodeExists for /hbase/master
    2012-12-02 03:42:48,001 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30002, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30003, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30001, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30002
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30003
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30001
    2012-12-02 03:42:48,182 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44800
    2012-12-02 03:42:48,194 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44800
    2012-12-02 03:42:48,196 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0003 with negotiated timeout 40000 for
    client /10.40.222.77:44800


    Regards,
    Shouvanik


    On Mon, Dec 3, 2012 at 1:44 AM, Harsh J wrote:

    Hi Shouvanik,

    Those logs are all INFO level and there's no issue in it. Can you
    share the issue/message you get when you mean "I am not able to start
    hbase" instead?

    On Mon, Dec 3, 2012 at 1:35 AM, Shouvanik Haldar
    wrote:
    Hi Harsh,

    I am not able to start hbase although hbase service is up. I am
    getting
    weird zookeeper exception. can you please help

    [root@ip-10-40-222-77 ~]# tail -f
    /var/log/zookeeper/zookeeper-cmf-zookeeper1-SERVER-ip-10-40-222-77.ec2.internal.log
    2012-12-02 03:42:47,991 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
    KeeperException when processing sessionid:0x13b5ac2ad7a0002
    type:create
    cxid:0x6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
    Path:/hbase/master Error:KeeperErrorCode = NodeExists for
    /hbase/master
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30002, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30003, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30001, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30002
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30003
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30001
    2012-12-02 03:42:48,182 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44800
    2012-12-02 03:42:48,194 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44800
    2012-12-02 03:42:48,196 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0003 with negotiated timeout 40000
    for
    client /10.40.222.77:44800


    Regards,
    Shouvanik Haldar


    On Mon, Dec 3, 2012 at 1:24 AM, Harsh J wrote:

    I replied to this over your duplicate post at

    https://groups.google.com/a/cloudera.org/d/topic/cdh-user/ft02DH52BZ4/discussion
    On Mon, Dec 3, 2012 at 12:59 AM, John H wrote:

    Hi,

    I installed Cloudera Manager(4.1.1) and CDH4 (4.1.2) using
    tarball
    on
    Ubuntu 10.04.4 (VM using virtual box). I was able to setup
    Cloudera
    manager
    server and agent, setup roles (NN,DN and Secondary NN) in pseudo
    distributed
    mode. I deployed the configuration to hosts and formatted name node
    successfully.

    When I started the services from Cloudera manager, I got message
    saying
    all
    3 (NN,DN,SNN) were started successfully, but I cannot access anyone
    of
    them
    through web url.
    The status for all 3 is started with good health.


    I looked at the log file for namenode and it is stuck at this line



    "2012-12-02 10:33:33,841 INFO
    org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager:
    Going
    to
    retain 2 images with txid >= 0"

    This is the first start after format, so I am confused why this is
    happening... Any help would be really appreciated..


    Thanks
    JH





    FULL LOG FILE: ***************************



    ************************************************************/
    2012-12-02 10:32:19,931 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting NameNode
    STARTUP_MSG: host = ubuntu1/127.0.0.1
    STARTUP_MSG: args = []
    STARTUP_MSG: version = 2.0.0-cdh4.1.2
    STARTUP_MSG: classpath =

    /home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-log4j12-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-json-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jettison-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jets3t-0.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-digester-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-annotations-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/activation-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/kfs-0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-net-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-auth-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsch-0.1.42.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-api-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/paranamer-2.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-math-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/junit-4.8.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/avro-1.7.1.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/stax-api-1.0.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-tests.jar:/home/john/cloudera-manager/cm-4.1.1/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-daemon-1.0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-tests.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-sources.jar:/usr/lib/hadoop-yarn/share/hadoop/yarn/*:/usr/lib/hadoop-mapreduce/share/hadoop/mapreduce/*
    STARTUP_MSG: build =

    file:///var/lib/jenkins/workspace/CDH4.1.2-Packaging-Hadoop/build/cdh4/hadoop/2.0.0-cdh4.1.2/source/hadoop-common-project/hadoop-common
    -r f0b53c81cbf56f5955e403b49fcd27afd5f082de; compiled by 'jenkins'
    on
    Thu
    Nov 1 17:06:55 PDT 2012
    ************************************************************/
    2012-12-02 10:32:22,214 INFO
    org.apache.hadoop.metrics2.impl.MetricsConfig:
    loaded properties from hadoop-metrics2.properties
    2012-12-02 10:32:23,240 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled
    snapshot
    period
    at 10 second(s).
    2012-12-02 10:32:23,240 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
    system
    started
    2012-12-02 10:32:25,269 INFO
    org.apache.hadoop.util.HostsFileReader:
    Refreshing hosts (include/exclude) list
    2012-12-02 10:32:25,270 INFO
    org.apache.hadoop.util.HostsFileReader:
    Adding
    127.0.0.1 to the list of hosts from

    /home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE/dfs_hosts_allow.txt
    2012-12-02 10:32:25,362 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
    dfs.block.invalidate.limit=1000
    2012-12-02 10:32:25,422 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    dfs.block.access.token.enable=false
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    defaultReplication = 1
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    maxReplication
    = 512
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    minReplication
    = 1
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    maxReplicationStreams = 2
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    shouldCheckForEnoughRacks = true
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    replicationRecheckInterval = 3000
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    encryptDataTransfer = false
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
    =
    john (auth:SIMPLE)
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
    =
    john
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    isPermissionEnabled
    =
    true
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled:
    false
    2012-12-02 10:32:25,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append
    Enabled:
    true
    2012-12-02 10:32:27,741 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
    occuring
    more than 10 times
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.threshold-pct = 0.9990000128746033
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.min.datanodes = 0
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.extension = 30000
    2012-12-02 10:32:27,941 INFO
    org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /home/john/dfs_dir/namenode/data1/in_use.lock acquired by
    nodename
    2070@ubuntu1
    2012-12-02 10:32:28,574 INFO
    org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /home/john/dfs_dir/namenode/data2/in_use.lock acquired by
    nodename
    2070@ubuntu1
    2012-12-02 10:32:28,803 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Recovering
    unfinalized segments in /home/john/dfs_dir/namenode/data1/current
    2012-12-02 10:32:28,803 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Recovering
    unfinalized segments in /home/john/dfs_dir/namenode/data2/current
    2012-12-02 10:32:28,805 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    No edit log streams selected.
    2012-12-02 10:32:28,950 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Loading image file
    /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000
    using
    no compression
    2012-12-02 10:32:28,950 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Number of files = 1
    2012-12-02 10:32:28,951 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Number of files under construction = 0
    2012-12-02 10:32:28,952 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Image file of size 117 loaded in 0 seconds.
    2012-12-02 10:32:28,952 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Loaded image for txid 0 from
    /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000
    2012-12-02 10:32:28,955 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
    segment
    at 1
    2012-12-02 10:32:30,041 INFO
    org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with
    entries
    0 lookups
    2012-12-02 10:32:30,041 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished
    loading
    FSImage in 2298 msecs
    2012-12-02 10:32:31,360 INFO org.apache.hadoop.ipc.Server: Starting
    Socket
    Reader #1 for port 8020
    2012-12-02 10:32:31,402 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
    FSNamesystemState MBean
    2012-12-02 10:32:31,482 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
    blocks
    under
    construction: 0
    2012-12-02 10:32:31,482 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: initializing
    replication queues
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Total
    number
    of
    blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number
    of
    invalid blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number
    of
    under-replicated blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number
    of
    over-replicated blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number
    of
    blocks
    being written = 0
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    Replication Queue initialization scan for invalid, over- and
    under-replicated blocks completed in 31 msec
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    Leaving safe mode after 6 secs.
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    Network topology has 0 racks and 0 datanodes
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    UnderReplicatedBlocks has 0 blocks
    2012-12-02 10:32:31,674 INFO org.mortbay.log: Logging to
    org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
    org.mortbay.log.Slf4jLog
    2012-12-02 10:32:31,814 INFO org.apache.hadoop.http.HttpServer:
    Added
    global
    filter 'safety'
    (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer:
    Added
    filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context hdfs
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer:
    Added
    filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context logs
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer:
    Added
    filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context static
    2012-12-02 10:32:31,841 INFO org.apache.hadoop.http.HttpServer:
    dfs.webhdfs.enabled = true
    2012-12-02 10:32:31,842 INFO org.apache.hadoop.http.HttpServer:
    Added
    filter
    'SPNEGO' (class=org.apache.hadoop.hdfs.web.AuthFilter)
    2012-12-02 10:32:31,844 INFO org.apache.hadoop.http.HttpServer:
    addJerseyResourcePackage:

    packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
    pathSpec=/webhdfs/v1/*
    2012-12-02 10:32:31,862 INFO org.apache.hadoop.http.HttpServer:
    Jetty
    bound
    to port 50070
    2012-12-02 10:32:31,862 INFO org.mortbay.log:
    jetty-6.1.26.cloudera.2
    2012-12-02 10:32:32,261 WARN
    org.apache.hadoop.security.authentication.server.AuthenticationFilter:
    'signature.secret' configuration not set, using a random value as
    secret
    2012-12-02 10:32:32,453 INFO org.mortbay.log: Started
    selectchannelconnector@localhost.localdomain:50070
    2012-12-02 10:32:32,453 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at:
    localhost.localdomain:50070
    2012-12-02 10:32:32,453 INFO org.apache.hadoop.ipc.Server: IPC
    Server
    Responder: starting
    2012-12-02 10:32:32,454 INFO org.apache.hadoop.ipc.Server: IPC
    Server
    listener on 8020: starting
    2012-12-02 10:32:32,603 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode up at:
    ubuntu1/127.0.0.1:8020
    2012-12-02 10:32:32,603 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting
    services
    required for active state
    2012-12-02 10:32:34,359 INFO
    org.apache.hadoop.fs.TrashPolicyDefault:
    The
    configured checkpoint interval is 0 minutes. Using an interval of
    1440
    minutes that is used for deletion instead
    2012-12-02 10:32:35,780 INFO org.apache.hadoop.hdfs.StateChange:
    BLOCK*
    NameSystem.registerDatanode: node registration from
    DatanodeRegistration(127.0.0.1,
    storageID=DS-377461142-127.0.0.1-50010-1354473155123,
    infoPort=50075,
    ipcPort=50020, storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0)
    storage
    DS-377461142-127.0.0.1-50010-1354473155123
    2012-12-02 10:32:35,790 INFO org.apache.hadoop.net.NetworkTopology:
    Adding a
    new node: /default/127.0.0.1:50010
    2012-12-02 10:32:35,951 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: BLOCK*
    processReport: Received first block report from 127.0.0.1:50010after
    becoming active. Its block contents are no longer considered stale.
    2012-12-02 10:32:35,951 INFO org.apache.hadoop.hdfs.StateChange:
    BLOCK*
    processReport: from DatanodeRegistration(127.0.0.1,
    storageID=DS-377461142-127.0.0.1-50010-1354473155123,
    infoPort=50075,
    ipcPort=50020, storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0),
    blocks:
    0, processing time: 1 msecs
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
    from
    127.0.0.1
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit
    logs.
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
    segment
    1
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
    transactions: 2
    Total time for transactions(ms): 1Number of transactions batched in
    Syncs: 0
    Number of syncs: 2 SyncTimes(ms): 166 189
    2012-12-02 10:33:31,595 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
    transactions: 2
    Total time for transactions(ms): 1Number of transactions batched in
    Syncs: 0
    Number of syncs: 3 SyncTimes(ms): 215 255
    2012-12-02 10:33:31,603 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Finalizing
    edits
    file

    /home/john/dfs_dir/namenode/data1/current/edits_inprogress_0000000000000000001
    ->

    /home/john/dfs_dir/namenode/data1/current/edits_0000000000000000001-0000000000000000002
    2012-12-02 10:33:31,604 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Finalizing
    edits
    file

    /home/john/dfs_dir/namenode/data2/current/edits_inprogress_0000000000000000001
    ->

    /home/john/dfs_dir/namenode/data2/current/edits_0000000000000000001-0000000000000000002
    2012-12-02 10:33:31,604 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
    segment
    at 3
    2012-12-02 10:33:33,274 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Opening
    connection
    to

    http://ubuntu1:50090/getimage?getimage=1&txid=2&storageInfo=-40:829593698:0:cluster2
    2012-12-02 10:33:33,622 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Transfer
    took
    0.34s
    at 0.00 KB/s
    2012-12-02 10:33:33,622 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Downloaded
    file
    fsimage.ckpt_0000000000000000002 size 117 bytes.
    2012-12-02 10:33:33,841 INFO
    org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager:
    Going
    to
    retain 2 images with txid >= 0



    --
    Harsh J



    --
    Thanks,
    Shouvanik


    --
    Harsh J



    --
    Thanks,
    Shouvanik


    --
    Harsh J


    --
    Thanks,
    *Shouvanik*
  • Harsh J at Dec 3, 2012 at 5:38 am
    Hi,

    Have you done the Generate Client Config steps as described in the CM
    3.7 manual here:
    https://ccp.cloudera.com/display/ENT/Generating+Client+Configuration?
    Without that, this is expected. Basically, your /etc/hbase/conf/ must
    be properly populated for clients.

    On Mon, Dec 3, 2012 at 10:59 AM, Shouvanik Haldar
    wrote:
    Hi Harsh,

    I performed the following steps as advised by you

    1. If there exists a /etc/zookeeper/zoo.cfg on your client machine,

    please rename it to something else, such as zoo.cfg.unwanted, or
    remove it altogether, and retry.


    2. Alternatively, try to run your HBase shell command(s) with the direct

    script (instead of wrapper /usr/bin/hbase) as:

    /usr/lib/hbase/bin/hbase shell

    .


    But, still I am getting the below error


    hbase(main):001:0> describe 't1'

    ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able
    to connect to ZooKeeper but the connection closes immediately. This could be
    a sign that the server has too many connections (30 is the default).
    Consider inspecting your ZK server logs for that error and then make sure
    you are reusing HBaseConfiguration as often as you can. See HTable's javadoc
    for more information.

    Here is some help for this command:
    Describe the named table. For example:
    hbase> describe 't1'


    Please help!
    Regards,
    Shouvanik

    On Mon, Dec 3, 2012 at 2:04 AM, Harsh J wrote:

    Hi Shouvanik,

    I notice you use CDH3. You should really be using CDH4, as it provides
    a great deal of improvements over the CDH3 version that you'll be
    missing out on.

    In any case:

    If there exists a /etc/zookeeper/zoo.cfg on your client machine,
    please rename it to something else, such as zoo.cfg.unwanted, or
    remove it altogether, and retry.

    Alternatively, try to run your HBase shell command(s) with the direct
    script (instead of wrapper /usr/bin/hbase) as:

    /usr/lib/hbase/bin/hbase shell

    And see if it lets you connect and work normally this time.

    On Mon, Dec 3, 2012 at 1:50 AM, Shouvanik Haldar
    wrote:
    Yes Harsh,

    [root@ip-10-40-222-77 ~]# hbase shell
    HBase Shell; enter 'help<RETURN>' for list of supported commands.
    Type "exit<RETURN>" to leave the HBase Shell
    Version 0.90.4-cdh3u3, r, Thu Jan 26 10:13:36 PST 2012

    hbase(main):001:0> describe 't1'

    ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is
    able
    to connect to ZooKeeper but the connection closes immediately. This
    could be
    a sign that the server has too many connections (30 is the default).
    Consider inspecting your ZK server logs for that error and then make
    sure
    you are reusing HBaseConfiguration as often as you can. See HTable's
    javadoc
    for more information.

    Here is some help for this command:
    Describe the named table. For example:
    hbase> describe 't1'


    hbase(main):002:0>


    Secondly,

    When I see the zookeeper logs, it says

    [root@ip-10-40-222-77 ~]# cat

    /var/log/zookeeper/zookeeper-cmf-zookeeper1-SERVER-ip-10-40-222-77.ec2.internal.log
    2012-12-01 23:30:35,726 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/1-zookeeper-server/zoo.cfg
    2012-12-01 23:30:35,731 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or
    no
    quorum defined in config, running in standalone mode
    2012-12-01 23:30:35,811 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/1-zookeeper-server/zoo.cfg
    2012-12-01 23:30:35,812 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-01 23:30:35,842 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server

    environment:java.class.path=/var/run/cloudera-scm-agent/process/1-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server

    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server

    environment:user.dir=/var/run/cloudera-scm-agent/process/1-zookeeper-server
    2012-12-01 23:30:35,851 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-01 23:30:35,851 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-01 23:30:35,851 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-01 23:30:35,891 INFO org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-01 23:30:35,907 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.0
    2012-12-01 23:30:35,941 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 28
    2012-12-01 23:38:21,260 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/9-zookeeper-server/zoo.cfg
    2012-12-01 23:38:21,265 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or
    no
    quorum defined in config, running in standalone mode
    2012-12-01 23:38:21,339 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/9-zookeeper-server/zoo.cfg
    2012-12-01 23:38:21,340 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server

    environment:java.class.path=/var/run/cloudera-scm-agent/process/9-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server

    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server

    environment:user.dir=/var/run/cloudera-scm-agent/process/9-zookeeper-server
    2012-12-01 23:38:21,366 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-01 23:38:21,366 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-01 23:38:21,366 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-01 23:38:21,399 INFO org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-01 23:38:21,415 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.28
    2012-12-01 23:38:21,430 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 28
    2012-12-01 23:43:49,121 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:45741
    2012-12-01 23:43:49,128 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:45741
    2012-12-01 23:43:49,130 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log
    file:
    log.29
    2012-12-01 23:43:49,172 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0000 with negotiated timeout 40000 for
    client /10.83.35.173:45741
    2012-12-01 23:43:52,894 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:59824
    2012-12-01 23:43:52,897 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:59824
    2012-12-01 23:43:52,899 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0001 with negotiated timeout 40000 for
    client /10.40.222.77:59824
    2012-12-01 23:43:52,993 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:59825
    2012-12-01 23:43:52,996 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:59825
    2012-12-01 23:43:52,998 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0002 with negotiated timeout 40000 for
    client /10.40.222.77:59825
    2012-12-02 02:48:13,910 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/34-zookeeper-server/zoo.cfg
    2012-12-02 02:48:13,914 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or
    no
    quorum defined in config, running in standalone mode
    2012-12-02 02:48:14,002 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/34-zookeeper-server/zoo.cfg
    2012-12-02 02:48:14,003 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-02 02:48:14,030 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-02 02:48:14,030 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-02 02:48:14,030 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server

    environment:java.class.path=/var/run/cloudera-scm-agent/process/34-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server

    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server

    environment:user.dir=/var/run/cloudera-scm-agent/process/34-zookeeper-server
    2012-12-02 02:48:14,039 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-02 02:48:14,039 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-02 02:48:14,040 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-02 02:48:14,072 INFO org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-02 02:48:14,088 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.28
    2012-12-02 02:48:14,109 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 2c
    2012-12-02 02:48:15,025 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:39458
    2012-12-02 02:48:15,029 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0002 at
    /10.40.222.77:39458
    2012-12-02 02:48:15,035 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0002 with negotiated timeout 40000 for
    client /10.40.222.77:39458
    2012-12-02 02:48:15,060 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:56157
    2012-12-02 02:48:15,062 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0000 at
    /10.83.35.173:56157
    2012-12-02 02:48:15,063 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0000 with negotiated timeout 40000 for
    client /10.83.35.173:56157
    2012-12-02 02:48:15,095 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:39459
    2012-12-02 02:48:15,095 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0001 at
    /10.40.222.77:39459
    2012-12-02 02:48:15,096 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0001 with negotiated timeout 40000 for
    client /10.40.222.77:39459
    2012-12-02 03:23:48,488 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log
    file:
    log.2d
    2012-12-02 03:23:48,560 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0001
    2012-12-02 03:23:48,570 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:39459 which had
    sessionid
    0x13b59e867fc0001
    2012-12-02 03:25:46,004 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54443
    2012-12-02 03:25:46,007 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54443
    2012-12-02 03:25:46,009 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30000 with negotiated timeout 40000 for
    client /10.40.222.77:54443
    2012-12-02 03:25:46,348 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client
    sessionid
    0x13b5a963ea30000, likely client has closed socket
    2012-12-02 03:25:46,349 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:54443 which had
    sessionid
    0x13b5a963ea30000
    2012-12-02 03:26:15,044 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client
    sessionid
    0x13b59e867fc0000, likely client has closed socket
    2012-12-02 03:26:15,045 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.83.35.173:56157 which had
    sessionid
    0x13b59e867fc0000
    2012-12-02 03:26:17,867 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client
    sessionid
    0x13b59e867fc0002, likely client has closed socket
    2012-12-02 03:26:17,867 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:39458 which had
    sessionid
    0x13b59e867fc0002
    2012-12-02 03:26:28,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30000, timeout of 40000ms exceeded
    2012-12-02 03:26:28,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30000
    2012-12-02 03:26:42,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b59e867fc0000, timeout of 40000ms exceeded
    2012-12-02 03:26:42,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0000
    2012-12-02 03:26:47,391 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54461
    2012-12-02 03:26:47,394 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54461
    2012-12-02 03:26:47,396 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30001 with negotiated timeout 40000 for
    client /10.40.222.77:54461
    2012-12-02 03:26:47,448 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54462
    2012-12-02 03:26:47,460 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54462
    2012-12-02 03:26:47,463 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30002 with negotiated timeout 40000 for
    client /10.40.222.77:54462
    2012-12-02 03:26:56,000 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b59e867fc0002, timeout of 40000ms exceeded
    2012-12-02 03:26:56,000 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0002
    2012-12-02 03:27:00,606 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:56794
    2012-12-02 03:27:00,614 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:56794
    2012-12-02 03:27:00,617 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30003 with negotiated timeout 40000 for
    client /10.83.35.173:56794
    2012-12-02 03:36:45,885 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/43-zookeeper-server/zoo.cfg
    2012-12-02 03:36:45,889 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or
    no
    quorum defined in config, running in standalone mode
    2012-12-02 03:36:45,962 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/43-zookeeper-server/zoo.cfg
    2012-12-02 03:36:45,962 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server

    environment:java.class.path=/var/run/cloudera-scm-agent/process/43-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server

    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server

    environment:user.dir=/var/run/cloudera-scm-agent/process/43-zookeeper-server
    2012-12-02 03:36:45,985 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-02 03:36:45,985 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-02 03:36:45,985 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-02 03:36:46,024 INFO org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-02 03:36:46,042 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.2c
    2012-12-02 03:36:46,063 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 36
    2012-12-02 03:36:46,531 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:37153
    2012-12-02 03:36:46,535 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30003 at
    /10.83.35.173:37153
    2012-12-02 03:36:46,539 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30003 with negotiated timeout 40000 for
    client /10.83.35.173:37153
    2012-12-02 03:36:46,883 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:47487
    2012-12-02 03:36:46,884 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30002 at
    /10.40.222.77:47487
    2012-12-02 03:36:46,884 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30002 with negotiated timeout 40000 for
    client /10.40.222.77:47487
    2012-12-02 03:36:47,310 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:47488
    2012-12-02 03:36:47,311 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30001 at
    /10.40.222.77:47488
    2012-12-02 03:36:47,312 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30001 with negotiated timeout 40000 for
    client /10.40.222.77:47488
    2012-12-02 03:41:46,616 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44776
    2012-12-02 03:41:46,619 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44776
    2012-12-02 03:41:46,620 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log
    file:
    log.37
    2012-12-02 03:41:46,674 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0000 with negotiated timeout 40000 for
    client /10.40.222.77:44776
    2012-12-02 03:41:47,494 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client
    sessionid
    0x13b5ac2ad7a0000, likely client has closed socket
    2012-12-02 03:41:47,495 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:44776 which had
    sessionid
    0x13b5ac2ad7a0000
    2012-12-02 03:42:16,001 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client
    sessionid
    0x13b5a963ea30003, likely client has closed socket
    2012-12-02 03:42:16,002 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.83.35.173:37153 which had
    sessionid
    0x13b5a963ea30003
    2012-12-02 03:42:17,916 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client
    sessionid
    0x13b5a963ea30002, likely client has closed socket
    2012-12-02 03:42:17,917 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:47487 which had
    sessionid
    0x13b5a963ea30002
    2012-12-02 03:42:19,468 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client
    sessionid
    0x13b5a963ea30001, likely client has closed socket
    2012-12-02 03:42:19,469 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:47488 which had
    sessionid
    0x13b5a963ea30001
    2012-12-02 03:42:28,000 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5ac2ad7a0000, timeout of 40000ms exceeded
    2012-12-02 03:42:28,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5ac2ad7a0000
    2012-12-02 03:42:46,214 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:52486
    2012-12-02 03:42:46,217 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:52486
    2012-12-02 03:42:46,220 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0001 with negotiated timeout 40000 for
    client /10.83.35.173:52486
    2012-12-02 03:42:47,857 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44799
    2012-12-02 03:42:47,864 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44799
    2012-12-02 03:42:47,867 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0002 with negotiated timeout 40000 for
    client /10.40.222.77:44799

    2012-12-02 03:42:47,991 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
    KeeperException when processing sessionid:0x13b5ac2ad7a0002 type:create
    cxid:0x6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
    Path:/hbase/master Error:KeeperErrorCode = NodeExists for /hbase/master
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30002, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30003, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30001, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30002
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30003
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30001
    2012-12-02 03:42:48,182 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44800
    2012-12-02 03:42:48,194 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44800
    2012-12-02 03:42:48,196 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0003 with negotiated timeout 40000 for
    client /10.40.222.77:44800


    Regards,
    Shouvanik


    On Mon, Dec 3, 2012 at 1:44 AM, Harsh J wrote:

    Hi Shouvanik,

    Those logs are all INFO level and there's no issue in it. Can you
    share the issue/message you get when you mean "I am not able to start
    hbase" instead?

    On Mon, Dec 3, 2012 at 1:35 AM, Shouvanik Haldar
    wrote:
    Hi Harsh,

    I am not able to start hbase although hbase service is up. I am
    getting
    weird zookeeper exception. can you please help

    [root@ip-10-40-222-77 ~]# tail -f


    /var/log/zookeeper/zookeeper-cmf-zookeeper1-SERVER-ip-10-40-222-77.ec2.internal.log
    2012-12-02 03:42:47,991 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
    KeeperException when processing sessionid:0x13b5ac2ad7a0002
    type:create
    cxid:0x6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
    Path:/hbase/master Error:KeeperErrorCode = NodeExists for
    /hbase/master
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30002, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30003, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30001, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30002
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30003
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30001
    2012-12-02 03:42:48,182 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44800
    2012-12-02 03:42:48,194 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44800
    2012-12-02 03:42:48,196 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0003 with negotiated timeout 40000
    for
    client /10.40.222.77:44800


    Regards,
    Shouvanik Haldar


    On Mon, Dec 3, 2012 at 1:24 AM, Harsh J wrote:

    I replied to this over your duplicate post at



    https://groups.google.com/a/cloudera.org/d/topic/cdh-user/ft02DH52BZ4/discussion

    On Mon, Dec 3, 2012 at 12:59 AM, John H <john.hadoop@gmail.com>
    wrote:
    Hi,

    I installed Cloudera Manager(4.1.1) and CDH4 (4.1.2) using
    tarball
    on
    Ubuntu 10.04.4 (VM using virtual box). I was able to setup
    Cloudera
    manager
    server and agent, setup roles (NN,DN and Secondary NN) in pseudo
    distributed
    mode. I deployed the configuration to hosts and formatted name
    node
    successfully.

    When I started the services from Cloudera manager, I got message
    saying
    all
    3 (NN,DN,SNN) were started successfully, but I cannot access
    anyone
    of
    them
    through web url.
    The status for all 3 is started with good health.


    I looked at the log file for namenode and it is stuck at this line



    "2012-12-02 10:33:33,841 INFO
    org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager:
    Going
    to
    retain 2 images with txid >= 0"

    This is the first start after format, so I am confused why this is
    happening... Any help would be really appreciated..


    Thanks
    JH





    FULL LOG FILE: ***************************



    ************************************************************/
    2012-12-02 10:32:19,931 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting NameNode
    STARTUP_MSG: host = ubuntu1/127.0.0.1
    STARTUP_MSG: args = []
    STARTUP_MSG: version = 2.0.0-cdh4.1.2
    STARTUP_MSG: classpath =



    /home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-log4j12-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-json-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jettison-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jets3t-0.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-digester-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-annotations-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/activation-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/kfs-0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-net-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-auth-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsch-0.1.42.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-api-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/paranamer-2.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-math-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/junit-4.8.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/avro-1.7.1.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/stax-api-1.0.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-tests.jar:/home/john/cloudera-manager/cm-4.1.1/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-daemon-1.0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-tests.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-sources.jar:/usr/lib/hadoop-yarn/share/hadoop/yarn/*:/usr/lib/hadoop-mapreduce/share/hadoop/mapreduce/*
    STARTUP_MSG: build =



    file:///var/lib/jenkins/workspace/CDH4.1.2-Packaging-Hadoop/build/cdh4/hadoop/2.0.0-cdh4.1.2/source/hadoop-common-project/hadoop-common
    -r f0b53c81cbf56f5955e403b49fcd27afd5f082de; compiled by 'jenkins'
    on
    Thu
    Nov 1 17:06:55 PDT 2012
    ************************************************************/
    2012-12-02 10:32:22,214 INFO
    org.apache.hadoop.metrics2.impl.MetricsConfig:
    loaded properties from hadoop-metrics2.properties
    2012-12-02 10:32:23,240 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled
    snapshot
    period
    at 10 second(s).
    2012-12-02 10:32:23,240 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode
    metrics
    system
    started
    2012-12-02 10:32:25,269 INFO
    org.apache.hadoop.util.HostsFileReader:
    Refreshing hosts (include/exclude) list
    2012-12-02 10:32:25,270 INFO
    org.apache.hadoop.util.HostsFileReader:
    Adding
    127.0.0.1 to the list of hosts from



    /home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE/dfs_hosts_allow.txt
    2012-12-02 10:32:25,362 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
    dfs.block.invalidate.limit=1000
    2012-12-02 10:32:25,422 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    dfs.block.access.token.enable=false
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    defaultReplication = 1
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    maxReplication
    = 512
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    minReplication
    = 1
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    maxReplicationStreams = 2
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    shouldCheckForEnoughRacks = true
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    replicationRecheckInterval = 3000
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    encryptDataTransfer = false
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
    =
    john (auth:SIMPLE)
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
    =
    john
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    isPermissionEnabled
    =
    true
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled:
    false
    2012-12-02 10:32:25,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append
    Enabled:
    true
    2012-12-02 10:32:27,741 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file
    names
    occuring
    more than 10 times
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.threshold-pct = 0.9990000128746033
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.min.datanodes = 0
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.extension = 30000
    2012-12-02 10:32:27,941 INFO
    org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /home/john/dfs_dir/namenode/data1/in_use.lock acquired by
    nodename
    2070@ubuntu1
    2012-12-02 10:32:28,574 INFO
    org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /home/john/dfs_dir/namenode/data2/in_use.lock acquired by
    nodename
    2070@ubuntu1
    2012-12-02 10:32:28,803 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Recovering
    unfinalized segments in /home/john/dfs_dir/namenode/data1/current
    2012-12-02 10:32:28,803 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Recovering
    unfinalized segments in /home/john/dfs_dir/namenode/data2/current
    2012-12-02 10:32:28,805 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    No edit log streams selected.
    2012-12-02 10:32:28,950 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Loading image file

    /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000
    using
    no compression
    2012-12-02 10:32:28,950 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Number of files = 1
    2012-12-02 10:32:28,951 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Number of files under construction = 0
    2012-12-02 10:32:28,952 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Image file of size 117 loaded in 0 seconds.
    2012-12-02 10:32:28,952 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Loaded image for txid 0 from

    /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000
    2012-12-02 10:32:28,955 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
    segment
    at 1
    2012-12-02 10:32:30,041 INFO
    org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with

    entries
    0 lookups
    2012-12-02 10:32:30,041 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished
    loading
    FSImage in 2298 msecs
    2012-12-02 10:32:31,360 INFO org.apache.hadoop.ipc.Server:
    Starting
    Socket
    Reader #1 for port 8020
    2012-12-02 10:32:31,402 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
    FSNamesystemState MBean
    2012-12-02 10:32:31,482 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
    blocks
    under
    construction: 0
    2012-12-02 10:32:31,482 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: initializing
    replication queues
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Total
    number
    of
    blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number
    of
    invalid blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number
    of
    under-replicated blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number
    of
    over-replicated blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number
    of
    blocks
    being written = 0
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    Replication Queue initialization scan for invalid, over- and
    under-replicated blocks completed in 31 msec
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    Leaving safe mode after 6 secs.
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    Network topology has 0 racks and 0 datanodes
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    UnderReplicatedBlocks has 0 blocks
    2012-12-02 10:32:31,674 INFO org.mortbay.log: Logging to
    org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
    org.mortbay.log.Slf4jLog
    2012-12-02 10:32:31,814 INFO org.apache.hadoop.http.HttpServer:
    Added
    global
    filter 'safety'
    (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer:
    Added
    filter
    static_user_filter


    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context hdfs
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer:
    Added
    filter
    static_user_filter


    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context logs
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer:
    Added
    filter
    static_user_filter


    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context static
    2012-12-02 10:32:31,841 INFO org.apache.hadoop.http.HttpServer:
    dfs.webhdfs.enabled = true
    2012-12-02 10:32:31,842 INFO org.apache.hadoop.http.HttpServer:
    Added
    filter
    'SPNEGO' (class=org.apache.hadoop.hdfs.web.AuthFilter)
    2012-12-02 10:32:31,844 INFO org.apache.hadoop.http.HttpServer:
    addJerseyResourcePackage:



    packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
    pathSpec=/webhdfs/v1/*
    2012-12-02 10:32:31,862 INFO org.apache.hadoop.http.HttpServer:
    Jetty
    bound
    to port 50070
    2012-12-02 10:32:31,862 INFO org.mortbay.log:
    jetty-6.1.26.cloudera.2
    2012-12-02 10:32:32,261 WARN


    org.apache.hadoop.security.authentication.server.AuthenticationFilter:
    'signature.secret' configuration not set, using a random value as
    secret
    2012-12-02 10:32:32,453 INFO org.mortbay.log: Started
    selectchannelconnector@localhost.localdomain:50070
    2012-12-02 10:32:32,453 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at:
    localhost.localdomain:50070
    2012-12-02 10:32:32,453 INFO org.apache.hadoop.ipc.Server: IPC
    Server
    Responder: starting
    2012-12-02 10:32:32,454 INFO org.apache.hadoop.ipc.Server: IPC
    Server
    listener on 8020: starting
    2012-12-02 10:32:32,603 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode up at:
    ubuntu1/127.0.0.1:8020
    2012-12-02 10:32:32,603 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting
    services
    required for active state
    2012-12-02 10:32:34,359 INFO
    org.apache.hadoop.fs.TrashPolicyDefault:
    The
    configured checkpoint interval is 0 minutes. Using an interval of
    1440
    minutes that is used for deletion instead
    2012-12-02 10:32:35,780 INFO org.apache.hadoop.hdfs.StateChange:
    BLOCK*
    NameSystem.registerDatanode: node registration from
    DatanodeRegistration(127.0.0.1,
    storageID=DS-377461142-127.0.0.1-50010-1354473155123,
    infoPort=50075,
    ipcPort=50020, storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0)
    storage
    DS-377461142-127.0.0.1-50010-1354473155123
    2012-12-02 10:32:35,790 INFO
    org.apache.hadoop.net.NetworkTopology:
    Adding a
    new node: /default/127.0.0.1:50010
    2012-12-02 10:32:35,951 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: BLOCK*
    processReport: Received first block report from 127.0.0.1:50010
    after
    becoming active. Its block contents are no longer considered
    stale.
    2012-12-02 10:32:35,951 INFO org.apache.hadoop.hdfs.StateChange:
    BLOCK*
    processReport: from DatanodeRegistration(127.0.0.1,
    storageID=DS-377461142-127.0.0.1-50010-1354473155123,
    infoPort=50075,
    ipcPort=50020,
    storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0),
    blocks:
    0, processing time: 1 msecs
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
    from
    127.0.0.1
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit
    logs.
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
    segment
    1
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
    transactions: 2
    Total time for transactions(ms): 1Number of transactions batched
    in
    Syncs: 0
    Number of syncs: 2 SyncTimes(ms): 166 189
    2012-12-02 10:33:31,595 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
    transactions: 2
    Total time for transactions(ms): 1Number of transactions batched
    in
    Syncs: 0
    Number of syncs: 3 SyncTimes(ms): 215 255
    2012-12-02 10:33:31,603 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Finalizing
    edits
    file



    /home/john/dfs_dir/namenode/data1/current/edits_inprogress_0000000000000000001
    ->



    /home/john/dfs_dir/namenode/data1/current/edits_0000000000000000001-0000000000000000002
    2012-12-02 10:33:31,604 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Finalizing
    edits
    file



    /home/john/dfs_dir/namenode/data2/current/edits_inprogress_0000000000000000001
    ->



    /home/john/dfs_dir/namenode/data2/current/edits_0000000000000000001-0000000000000000002
    2012-12-02 10:33:31,604 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
    segment
    at 3
    2012-12-02 10:33:33,274 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Opening
    connection
    to



    http://ubuntu1:50090/getimage?getimage=1&txid=2&storageInfo=-40:829593698:0:cluster2
    2012-12-02 10:33:33,622 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Transfer
    took
    0.34s
    at 0.00 KB/s
    2012-12-02 10:33:33,622 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Downloaded
    file
    fsimage.ckpt_0000000000000000002 size 117 bytes.
    2012-12-02 10:33:33,841 INFO
    org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager:
    Going
    to
    retain 2 images with txid >= 0



    --
    Harsh J



    --
    Thanks,
    Shouvanik


    --
    Harsh J



    --
    Thanks,
    Shouvanik


    --
    Harsh J



    --
    Thanks,
    Shouvanik


    --
    Harsh J
  • Shouvanik Haldar at Dec 3, 2012 at 5:47 am
    I did the step "Generate Client Config".
    But I still getting ZK exception. Tell me onething, do I have to start ZK
    after these changes or I just have to try to connect to hbase after these
    steps?

    Please help me
    Regards,
    Shouvanik
    On Mon, Dec 3, 2012 at 11:07 AM, Harsh J wrote:

    Hi,

    Have you done the Generate Client Config steps as described in the CM
    3.7 manual here:
    https://ccp.cloudera.com/display/ENT/Generating+Client+Configuration?
    Without that, this is expected. Basically, your /etc/hbase/conf/ must
    be properly populated for clients.

    On Mon, Dec 3, 2012 at 10:59 AM, Shouvanik Haldar
    wrote:
    Hi Harsh,

    I performed the following steps as advised by you

    1. If there exists a /etc/zookeeper/zoo.cfg on your client machine,

    please rename it to something else, such as zoo.cfg.unwanted, or
    remove it altogether, and retry.


    2. Alternatively, try to run your HBase shell command(s) with the direct

    script (instead of wrapper /usr/bin/hbase) as:

    /usr/lib/hbase/bin/hbase shell

    .


    But, still I am getting the below error


    hbase(main):001:0> describe 't1'

    ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able
    to connect to ZooKeeper but the connection closes immediately. This could be
    a sign that the server has too many connections (30 is the default).
    Consider inspecting your ZK server logs for that error and then make sure
    you are reusing HBaseConfiguration as often as you can. See HTable's javadoc
    for more information.

    Here is some help for this command:
    Describe the named table. For example:
    hbase> describe 't1'


    Please help!
    Regards,
    Shouvanik

    On Mon, Dec 3, 2012 at 2:04 AM, Harsh J wrote:

    Hi Shouvanik,

    I notice you use CDH3. You should really be using CDH4, as it provides
    a great deal of improvements over the CDH3 version that you'll be
    missing out on.

    In any case:

    If there exists a /etc/zookeeper/zoo.cfg on your client machine,
    please rename it to something else, such as zoo.cfg.unwanted, or
    remove it altogether, and retry.

    Alternatively, try to run your HBase shell command(s) with the direct
    script (instead of wrapper /usr/bin/hbase) as:

    /usr/lib/hbase/bin/hbase shell

    And see if it lets you connect and work normally this time.

    On Mon, Dec 3, 2012 at 1:50 AM, Shouvanik Haldar
    wrote:
    Yes Harsh,

    [root@ip-10-40-222-77 ~]# hbase shell
    HBase Shell; enter 'help<RETURN>' for list of supported commands.
    Type "exit<RETURN>" to leave the HBase Shell
    Version 0.90.4-cdh3u3, r, Thu Jan 26 10:13:36 PST 2012

    hbase(main):001:0> describe 't1'

    ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is
    able
    to connect to ZooKeeper but the connection closes immediately. This
    could be
    a sign that the server has too many connections (30 is the default).
    Consider inspecting your ZK server logs for that error and then make
    sure
    you are reusing HBaseConfiguration as often as you can. See HTable's
    javadoc
    for more information.

    Here is some help for this command:
    Describe the named table. For example:
    hbase> describe 't1'


    hbase(main):002:0>


    Secondly,

    When I see the zookeeper logs, it says

    [root@ip-10-40-222-77 ~]# cat
    /var/log/zookeeper/zookeeper-cmf-zookeeper1-SERVER-ip-10-40-222-77.ec2.internal.log
    2012-12-01 23:30:35,726 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/1-zookeeper-server/zoo.cfg
    2012-12-01 23:30:35,731 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or
    no
    quorum defined in config, running in standalone mode
    2012-12-01 23:30:35,811 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/1-zookeeper-server/zoo.cfg
    2012-12-01 23:30:35,812 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-01 23:30:35,842 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/1-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/1-zookeeper-server
    2012-12-01 23:30:35,851 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-01 23:30:35,851 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-01 23:30:35,851 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-01 23:30:35,891 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-01 23:30:35,907 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.0
    2012-12-01 23:30:35,941 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting:
    28
    2012-12-01 23:38:21,260 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/9-zookeeper-server/zoo.cfg
    2012-12-01 23:38:21,265 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or
    no
    quorum defined in config, running in standalone mode
    2012-12-01 23:38:21,339 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/9-zookeeper-server/zoo.cfg
    2012-12-01 23:38:21,340 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/9-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/9-zookeeper-server
    2012-12-01 23:38:21,366 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-01 23:38:21,366 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-01 23:38:21,366 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-01 23:38:21,399 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-01 23:38:21,415 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.28
    2012-12-01 23:38:21,430 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting:
    28
    2012-12-01 23:43:49,121 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:45741
    2012-12-01 23:43:49,128 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:45741
    2012-12-01 23:43:49,130 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log
    file:
    log.29
    2012-12-01 23:43:49,172 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0000 with negotiated timeout 40000
    for
    client /10.83.35.173:45741
    2012-12-01 23:43:52,894 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:59824
    2012-12-01 23:43:52,897 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:59824
    2012-12-01 23:43:52,899 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0001 with negotiated timeout 40000
    for
    client /10.40.222.77:59824
    2012-12-01 23:43:52,993 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:59825
    2012-12-01 23:43:52,996 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:59825
    2012-12-01 23:43:52,998 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0002 with negotiated timeout 40000
    for
    client /10.40.222.77:59825
    2012-12-02 02:48:13,910 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/34-zookeeper-server/zoo.cfg
    2012-12-02 02:48:13,914 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or
    no
    quorum defined in config, running in standalone mode
    2012-12-02 02:48:14,002 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/34-zookeeper-server/zoo.cfg
    2012-12-02 02:48:14,003 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-02 02:48:14,030 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-02 02:48:14,030 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-02 02:48:14,030 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/34-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/34-zookeeper-server
    2012-12-02 02:48:14,039 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-02 02:48:14,039 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-02 02:48:14,040 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-02 02:48:14,072 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-02 02:48:14,088 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.28
    2012-12-02 02:48:14,109 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting:
    2c
    2012-12-02 02:48:15,025 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:39458
    2012-12-02 02:48:15,029 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0002 at
    /10.40.222.77:39458
    2012-12-02 02:48:15,035 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0002 with negotiated timeout 40000
    for
    client /10.40.222.77:39458
    2012-12-02 02:48:15,060 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:56157
    2012-12-02 02:48:15,062 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0000 at
    /10.83.35.173:56157
    2012-12-02 02:48:15,063 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0000 with negotiated timeout 40000
    for
    client /10.83.35.173:56157
    2012-12-02 02:48:15,095 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:39459
    2012-12-02 02:48:15,095 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0001 at
    /10.40.222.77:39459
    2012-12-02 02:48:15,096 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0001 with negotiated timeout 40000
    for
    client /10.40.222.77:39459
    2012-12-02 03:23:48,488 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log
    file:
    log.2d
    2012-12-02 03:23:48,560 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0001
    2012-12-02 03:23:48,570 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:39459 which had
    sessionid
    0x13b59e867fc0001
    2012-12-02 03:25:46,004 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54443
    2012-12-02 03:25:46,007 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54443
    2012-12-02 03:25:46,009 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30000 with negotiated timeout 40000
    for
    client /10.40.222.77:54443
    2012-12-02 03:25:46,348 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client
    sessionid
    0x13b5a963ea30000, likely client has closed socket
    2012-12-02 03:25:46,349 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:54443 which had
    sessionid
    0x13b5a963ea30000
    2012-12-02 03:26:15,044 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client
    sessionid
    0x13b59e867fc0000, likely client has closed socket
    2012-12-02 03:26:15,045 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.83.35.173:56157 which had
    sessionid
    0x13b59e867fc0000
    2012-12-02 03:26:17,867 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client
    sessionid
    0x13b59e867fc0002, likely client has closed socket
    2012-12-02 03:26:17,867 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:39458 which had
    sessionid
    0x13b59e867fc0002
    2012-12-02 03:26:28,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30000, timeout of 40000ms exceeded
    2012-12-02 03:26:28,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30000
    2012-12-02 03:26:42,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b59e867fc0000, timeout of 40000ms exceeded
    2012-12-02 03:26:42,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0000
    2012-12-02 03:26:47,391 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54461
    2012-12-02 03:26:47,394 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54461
    2012-12-02 03:26:47,396 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30001 with negotiated timeout 40000
    for
    client /10.40.222.77:54461
    2012-12-02 03:26:47,448 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54462
    2012-12-02 03:26:47,460 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54462
    2012-12-02 03:26:47,463 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30002 with negotiated timeout 40000
    for
    client /10.40.222.77:54462
    2012-12-02 03:26:56,000 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b59e867fc0002, timeout of 40000ms exceeded
    2012-12-02 03:26:56,000 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0002
    2012-12-02 03:27:00,606 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:56794
    2012-12-02 03:27:00,614 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:56794
    2012-12-02 03:27:00,617 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30003 with negotiated timeout 40000
    for
    client /10.83.35.173:56794
    2012-12-02 03:36:45,885 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/43-zookeeper-server/zoo.cfg
    2012-12-02 03:36:45,889 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or
    no
    quorum defined in config, running in standalone mode
    2012-12-02 03:36:45,962 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/43-zookeeper-server/zoo.cfg
    2012-12-02 03:36:45,962 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/43-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/43-zookeeper-server
    2012-12-02 03:36:45,985 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-02 03:36:45,985 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-02 03:36:45,985 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-02 03:36:46,024 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-02 03:36:46,042 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.2c
    2012-12-02 03:36:46,063 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting:
    36
    2012-12-02 03:36:46,531 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:37153
    2012-12-02 03:36:46,535 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30003 at
    /10.83.35.173:37153
    2012-12-02 03:36:46,539 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30003 with negotiated timeout 40000
    for
    client /10.83.35.173:37153
    2012-12-02 03:36:46,883 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:47487
    2012-12-02 03:36:46,884 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30002 at
    /10.40.222.77:47487
    2012-12-02 03:36:46,884 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30002 with negotiated timeout 40000
    for
    client /10.40.222.77:47487
    2012-12-02 03:36:47,310 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:47488
    2012-12-02 03:36:47,311 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30001 at
    /10.40.222.77:47488
    2012-12-02 03:36:47,312 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30001 with negotiated timeout 40000
    for
    client /10.40.222.77:47488
    2012-12-02 03:41:46,616 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44776
    2012-12-02 03:41:46,619 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44776
    2012-12-02 03:41:46,620 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log
    file:
    log.37
    2012-12-02 03:41:46,674 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0000 with negotiated timeout 40000
    for
    client /10.40.222.77:44776
    2012-12-02 03:41:47,494 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client
    sessionid
    0x13b5ac2ad7a0000, likely client has closed socket
    2012-12-02 03:41:47,495 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:44776 which had
    sessionid
    0x13b5ac2ad7a0000
    2012-12-02 03:42:16,001 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client
    sessionid
    0x13b5a963ea30003, likely client has closed socket
    2012-12-02 03:42:16,002 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.83.35.173:37153 which had
    sessionid
    0x13b5a963ea30003
    2012-12-02 03:42:17,916 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client
    sessionid
    0x13b5a963ea30002, likely client has closed socket
    2012-12-02 03:42:17,917 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:47487 which had
    sessionid
    0x13b5a963ea30002
    2012-12-02 03:42:19,468 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client
    sessionid
    0x13b5a963ea30001, likely client has closed socket
    2012-12-02 03:42:19,469 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:47488 which had
    sessionid
    0x13b5a963ea30001
    2012-12-02 03:42:28,000 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5ac2ad7a0000, timeout of 40000ms exceeded
    2012-12-02 03:42:28,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5ac2ad7a0000
    2012-12-02 03:42:46,214 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:52486
    2012-12-02 03:42:46,217 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:52486
    2012-12-02 03:42:46,220 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0001 with negotiated timeout 40000
    for
    client /10.83.35.173:52486
    2012-12-02 03:42:47,857 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44799
    2012-12-02 03:42:47,864 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44799
    2012-12-02 03:42:47,867 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0002 with negotiated timeout 40000
    for
    client /10.40.222.77:44799

    2012-12-02 03:42:47,991 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
    KeeperException when processing sessionid:0x13b5ac2ad7a0002
    type:create
    cxid:0x6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
    Path:/hbase/master Error:KeeperErrorCode = NodeExists for
    /hbase/master
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30002, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30003, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30001, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30002
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30003
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30001
    2012-12-02 03:42:48,182 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44800
    2012-12-02 03:42:48,194 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44800
    2012-12-02 03:42:48,196 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0003 with negotiated timeout 40000
    for
    client /10.40.222.77:44800


    Regards,
    Shouvanik


    On Mon, Dec 3, 2012 at 1:44 AM, Harsh J wrote:

    Hi Shouvanik,

    Those logs are all INFO level and there's no issue in it. Can you
    share the issue/message you get when you mean "I am not able to start
    hbase" instead?

    On Mon, Dec 3, 2012 at 1:35 AM, Shouvanik Haldar
    wrote:
    Hi Harsh,

    I am not able to start hbase although hbase service is up. I am
    getting
    weird zookeeper exception. can you please help

    [root@ip-10-40-222-77 ~]# tail -f

    /var/log/zookeeper/zookeeper-cmf-zookeeper1-SERVER-ip-10-40-222-77.ec2.internal.log
    2012-12-02 03:42:47,991 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
    KeeperException when processing sessionid:0x13b5ac2ad7a0002
    type:create
    cxid:0x6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
    Path:/hbase/master Error:KeeperErrorCode = NodeExists for
    /hbase/master
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30002, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30003, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30001, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30002
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30003
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30001
    2012-12-02 03:42:48,182 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44800
    2012-12-02 03:42:48,194 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44800
    2012-12-02 03:42:48,196 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0003 with negotiated timeout 40000
    for
    client /10.40.222.77:44800


    Regards,
    Shouvanik Haldar


    On Mon, Dec 3, 2012 at 1:24 AM, Harsh J wrote:

    I replied to this over your duplicate post at


    https://groups.google.com/a/cloudera.org/d/topic/cdh-user/ft02DH52BZ4/discussion
    On Mon, Dec 3, 2012 at 12:59 AM, John H <john.hadoop@gmail.com>
    wrote:
    Hi,

    I installed Cloudera Manager(4.1.1) and CDH4 (4.1.2) using
    tarball
    on
    Ubuntu 10.04.4 (VM using virtual box). I was able to setup
    Cloudera
    manager
    server and agent, setup roles (NN,DN and Secondary NN) in pseudo
    distributed
    mode. I deployed the configuration to hosts and formatted name
    node
    successfully.

    When I started the services from Cloudera manager, I got message
    saying
    all
    3 (NN,DN,SNN) were started successfully, but I cannot access
    anyone
    of
    them
    through web url.
    The status for all 3 is started with good health.


    I looked at the log file for namenode and it is stuck at this
    line


    "2012-12-02 10:33:33,841 INFO
    org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager:
    Going
    to
    retain 2 images with txid >= 0"

    This is the first start after format, so I am confused why this
    is
    happening... Any help would be really appreciated..


    Thanks
    JH





    FULL LOG FILE: ***************************



    ************************************************************/
    2012-12-02 10:32:19,931 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting NameNode
    STARTUP_MSG: host = ubuntu1/127.0.0.1
    STARTUP_MSG: args = []
    STARTUP_MSG: version = 2.0.0-cdh4.1.2
    STARTUP_MSG: classpath =


    /home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-log4j12-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-json-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jettison-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jets3t-0.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-digester-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-annotations-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/activation-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/kfs-0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-net-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-auth-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsch-0.1.42.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-api-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/paranamer-2.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-math-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/junit-4.8.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/avro-1.7.1.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/stax-api-1.0.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-tests.jar:/home/john/cloudera-manager/cm-4.1.1/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-daemon-1.0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-tests.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-sources.jar:/usr/lib/hadoop-yarn/share/hadoop/yarn/*:/usr/lib/hadoop-mapreduce/share/hadoop/mapreduce/*
    STARTUP_MSG: build =


    file:///var/lib/jenkins/workspace/CDH4.1.2-Packaging-Hadoop/build/cdh4/hadoop/2.0.0-cdh4.1.2/source/hadoop-common-project/hadoop-common
    -r f0b53c81cbf56f5955e403b49fcd27afd5f082de; compiled by
    'jenkins'
    on
    Thu
    Nov 1 17:06:55 PDT 2012
    ************************************************************/
    2012-12-02 10:32:22,214 INFO
    org.apache.hadoop.metrics2.impl.MetricsConfig:
    loaded properties from hadoop-metrics2.properties
    2012-12-02 10:32:23,240 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled
    snapshot
    period
    at 10 second(s).
    2012-12-02 10:32:23,240 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode
    metrics
    system
    started
    2012-12-02 10:32:25,269 INFO
    org.apache.hadoop.util.HostsFileReader:
    Refreshing hosts (include/exclude) list
    2012-12-02 10:32:25,270 INFO
    org.apache.hadoop.util.HostsFileReader:
    Adding
    127.0.0.1 to the list of hosts from


    /home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE/dfs_hosts_allow.txt
    2012-12-02 10:32:25,362 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
    dfs.block.invalidate.limit=1000
    2012-12-02 10:32:25,422 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    dfs.block.access.token.enable=false
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    defaultReplication = 1
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    maxReplication
    = 512
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    minReplication
    = 1
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    maxReplicationStreams = 2
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    shouldCheckForEnoughRacks = true
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    replicationRecheckInterval = 3000
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    encryptDataTransfer = false
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
    =
    john (auth:SIMPLE)
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
    =
    john
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    isPermissionEnabled
    =
    true
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled:
    false
    2012-12-02 10:32:25,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append
    Enabled:
    true
    2012-12-02 10:32:27,741 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file
    names
    occuring
    more than 10 times
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.threshold-pct = 0.9990000128746033
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.min.datanodes = 0
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.extension = 30000
    2012-12-02 10:32:27,941 INFO
    org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /home/john/dfs_dir/namenode/data1/in_use.lock acquired
    by
    nodename
    2070@ubuntu1
    2012-12-02 10:32:28,574 INFO
    org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /home/john/dfs_dir/namenode/data2/in_use.lock acquired
    by
    nodename
    2070@ubuntu1
    2012-12-02 10:32:28,803 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Recovering
    unfinalized segments in
    /home/john/dfs_dir/namenode/data1/current
    2012-12-02 10:32:28,803 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Recovering
    unfinalized segments in
    /home/john/dfs_dir/namenode/data2/current
    2012-12-02 10:32:28,805 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    No edit log streams selected.
    2012-12-02 10:32:28,950 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Loading image file
    /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000
    using
    no compression
    2012-12-02 10:32:28,950 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Number of files = 1
    2012-12-02 10:32:28,951 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Number of files under construction = 0
    2012-12-02 10:32:28,952 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Image file of size 117 loaded in 0 seconds.
    2012-12-02 10:32:28,952 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Loaded image for txid 0 from
    /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000
    2012-12-02 10:32:28,955 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
    segment
    at 1
    2012-12-02 10:32:30,041 INFO
    org.apache.hadoop.hdfs.server.namenode.NameCache: initialized
    with
    entries
    0 lookups
    2012-12-02 10:32:30,041 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished
    loading
    FSImage in 2298 msecs
    2012-12-02 10:32:31,360 INFO org.apache.hadoop.ipc.Server:
    Starting
    Socket
    Reader #1 for port 8020
    2012-12-02 10:32:31,402 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
    FSNamesystemState MBean
    2012-12-02 10:32:31,482 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
    blocks
    under
    construction: 0
    2012-12-02 10:32:31,482 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    initializing
    replication queues
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Total
    number
    of
    blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number
    of
    invalid blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number
    of
    under-replicated blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number
    of
    over-replicated blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number
    of
    blocks
    being written = 0
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    Replication Queue initialization scan for invalid, over- and
    under-replicated blocks completed in 31 msec
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    Leaving safe mode after 6 secs.
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    Network topology has 0 racks and 0 datanodes
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    UnderReplicatedBlocks has 0 blocks
    2012-12-02 10:32:31,674 INFO org.mortbay.log: Logging to
    org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
    org.mortbay.log.Slf4jLog
    2012-12-02 10:32:31,814 INFO org.apache.hadoop.http.HttpServer:
    Added
    global
    filter 'safety'
    (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer:
    Added
    filter
    static_user_filter

    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context hdfs
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer:
    Added
    filter
    static_user_filter

    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context logs
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer:
    Added
    filter
    static_user_filter

    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context static
    2012-12-02 10:32:31,841 INFO org.apache.hadoop.http.HttpServer:
    dfs.webhdfs.enabled = true
    2012-12-02 10:32:31,842 INFO org.apache.hadoop.http.HttpServer:
    Added
    filter
    'SPNEGO' (class=org.apache.hadoop.hdfs.web.AuthFilter)
    2012-12-02 10:32:31,844 INFO org.apache.hadoop.http.HttpServer:
    addJerseyResourcePackage:


    packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
    pathSpec=/webhdfs/v1/*
    2012-12-02 10:32:31,862 INFO org.apache.hadoop.http.HttpServer:
    Jetty
    bound
    to port 50070
    2012-12-02 10:32:31,862 INFO org.mortbay.log:
    jetty-6.1.26.cloudera.2
    2012-12-02 10:32:32,261 WARN

    org.apache.hadoop.security.authentication.server.AuthenticationFilter:
    'signature.secret' configuration not set, using a random value
    as
    secret
    2012-12-02 10:32:32,453 INFO org.mortbay.log: Started
    selectchannelconnector@localhost.localdomain:50070
    2012-12-02 10:32:32,453 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up
    at:
    localhost.localdomain:50070
    2012-12-02 10:32:32,453 INFO org.apache.hadoop.ipc.Server: IPC
    Server
    Responder: starting
    2012-12-02 10:32:32,454 INFO org.apache.hadoop.ipc.Server: IPC
    Server
    listener on 8020: starting
    2012-12-02 10:32:32,603 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode up at:
    ubuntu1/127.0.0.1:8020
    2012-12-02 10:32:32,603 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting
    services
    required for active state
    2012-12-02 10:32:34,359 INFO
    org.apache.hadoop.fs.TrashPolicyDefault:
    The
    configured checkpoint interval is 0 minutes. Using an interval
    of
    1440
    minutes that is used for deletion instead
    2012-12-02 10:32:35,780 INFO org.apache.hadoop.hdfs.StateChange:
    BLOCK*
    NameSystem.registerDatanode: node registration from
    DatanodeRegistration(127.0.0.1,
    storageID=DS-377461142-127.0.0.1-50010-1354473155123,
    infoPort=50075,
    ipcPort=50020,
    storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0)
    storage
    DS-377461142-127.0.0.1-50010-1354473155123
    2012-12-02 10:32:35,790 INFO
    org.apache.hadoop.net.NetworkTopology:
    Adding a
    new node: /default/127.0.0.1:50010
    2012-12-02 10:32:35,951 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    BLOCK*
    processReport: Received first block report from 127.0.0.1:50010
    after
    becoming active. Its block contents are no longer considered
    stale.
    2012-12-02 10:32:35,951 INFO org.apache.hadoop.hdfs.StateChange:
    BLOCK*
    processReport: from DatanodeRegistration(127.0.0.1,
    storageID=DS-377461142-127.0.0.1-50010-1354473155123,
    infoPort=50075,
    ipcPort=50020,
    storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0),
    blocks:
    0, processing time: 1 msecs
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit
    Log
    from
    127.0.0.1
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit
    logs.
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
    segment
    1
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
    transactions: 2
    Total time for transactions(ms): 1Number of transactions batched
    in
    Syncs: 0
    Number of syncs: 2 SyncTimes(ms): 166 189
    2012-12-02 10:33:31,595 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
    transactions: 2
    Total time for transactions(ms): 1Number of transactions batched
    in
    Syncs: 0
    Number of syncs: 3 SyncTimes(ms): 215 255
    2012-12-02 10:33:31,603 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Finalizing
    edits
    file


    /home/john/dfs_dir/namenode/data1/current/edits_inprogress_0000000000000000001
    ->


    /home/john/dfs_dir/namenode/data1/current/edits_0000000000000000001-0000000000000000002
    2012-12-02 10:33:31,604 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Finalizing
    edits
    file


    /home/john/dfs_dir/namenode/data2/current/edits_inprogress_0000000000000000001
    ->


    /home/john/dfs_dir/namenode/data2/current/edits_0000000000000000001-0000000000000000002
    2012-12-02 10:33:31,604 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
    segment
    at 3
    2012-12-02 10:33:33,274 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Opening
    connection
    to


    http://ubuntu1:50090/getimage?getimage=1&txid=2&storageInfo=-40:829593698:0:cluster2
    2012-12-02 10:33:33,622 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Transfer
    took
    0.34s
    at 0.00 KB/s
    2012-12-02 10:33:33,622 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage:
    Downloaded
    file
    fsimage.ckpt_0000000000000000002 size 117 bytes.
    2012-12-02 10:33:33,841 INFO
    org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager:
    Going
    to
    retain 2 images with txid >= 0



    --
    Harsh J



    --
    Thanks,
    Shouvanik


    --
    Harsh J



    --
    Thanks,
    Shouvanik


    --
    Harsh J



    --
    Thanks,
    Shouvanik


    --
    Harsh J


    --
    Thanks,
    *Shouvanik*
  • bc Wong at Dec 3, 2012 at 6:46 am

    On Sun, Dec 2, 2012 at 12:34 PM, Harsh J wrote:

    Hi Shouvanik,

    I notice you use CDH3. You should really be using CDH4, as it provides
    a great deal of improvements over the CDH3 version that you'll be
    missing out on.

    In any case:

    If there exists a /etc/zookeeper/zoo.cfg on your client machine,
    please rename it to something else, such as zoo.cfg.unwanted, or
    remove it altogether, and retry.

    Alternatively, try to run your HBase shell command(s) with the direct
    script (instead of wrapper /usr/bin/hbase) as:

    /usr/lib/hbase/bin/hbase shell

    And see if it lets you connect and work normally this time.

    On Mon, Dec 3, 2012 at 1:50 AM, Shouvanik Haldar
    wrote:
    Yes Harsh,

    [root@ip-10-40-222-77 ~]# hbase shell
    HBase Shell; enter 'help<RETURN>' for list of supported commands.
    Type "exit<RETURN>" to leave the HBase Shell
    Version 0.90.4-cdh3u3, r, Thu Jan 26 10:13:36 PST 2012

    hbase(main):001:0> describe 't1'

    ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able
    to connect to ZooKeeper but the connection closes immediately. This could be
    a sign that the server has too many connections (30 is the default).
    Consider inspecting your ZK server logs for that error and then make sure
    you are reusing HBaseConfiguration as often as you can. See HTable's javadoc
    for more information.

    Here is some help for this command:
    Describe the named table. For example:
    hbase> describe 't1'


    hbase(main):002:0>


    Secondly,

    When I see the zookeeper logs, it says

    [root@ip-10-40-222-77 ~]# cat
    /var/log/zookeeper/zookeeper-cmf-zookeeper1-SERVER-ip-10-40-222-77.ec2.internal.log
    2012-12-01 23:30:35,726 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/1-zookeeper-server/zoo.cfg
    2012-12-01 23:30:35,731 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-01 23:30:35,811 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/1-zookeeper-server/zoo.cfg
    2012-12-01 23:30:35,812 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-01 23:30:35,842 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on 01/26/2012
    20:08 GMT
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/1-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-01 23:30:35,843 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/1-zookeeper-server
    2012-12-01 23:30:35,851 INFO org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-01 23:30:35,851 INFO org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-01 23:30:35,851 INFO org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-01 23:30:35,891 INFO org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-01 23:30:35,907 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.0
    2012-12-01 23:30:35,941 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 28
    2012-12-01 23:38:21,260 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/9-zookeeper-server/zoo.cfg
    2012-12-01 23:38:21,265 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-01 23:38:21,339 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/9-zookeeper-server/zoo.cfg
    2012-12-01 23:38:21,340 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-01 23:38:21,357 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on 01/26/2012
    20:08 GMT
    2012-12-01 23:38:21,357 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-01 23:38:21,357 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-01 23:38:21,357 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-01 23:38:21,357 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-01 23:38:21,357 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/9-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-01 23:38:21,357 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-01 23:38:21,357 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-01 23:38:21,357 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-01 23:38:21,358 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-01 23:38:21,358 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-01 23:38:21,358 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-01 23:38:21,358 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-01 23:38:21,358 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-01 23:38:21,358 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/9-zookeeper-server
    2012-12-01 23:38:21,366 INFO org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-01 23:38:21,366 INFO org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-01 23:38:21,366 INFO org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-01 23:38:21,399 INFO org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-01 23:38:21,415 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.28
    2012-12-01 23:38:21,430 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 28
    2012-12-01 23:43:49,121 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:45741
    2012-12-01 23:43:49,128 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:45741
    2012-12-01 23:43:49,130 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.29
    2012-12-01 23:43:49,172 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0000 with negotiated timeout 40000 for
    client /10.83.35.173:45741
    2012-12-01 23:43:52,894 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:59824
    2012-12-01 23:43:52,897 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:59824
    2012-12-01 23:43:52,899 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0001 with negotiated timeout 40000 for
    client /10.40.222.77:59824
    2012-12-01 23:43:52,993 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:59825
    2012-12-01 23:43:52,996 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:59825
    2012-12-01 23:43:52,998 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0002 with negotiated timeout 40000 for
    client /10.40.222.77:59825
    2012-12-02 02:48:13,910 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/34-zookeeper-server/zoo.cfg
    2012-12-02 02:48:13,914 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-02 02:48:14,002 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/34-zookeeper-server/zoo.cfg
    2012-12-02 02:48:14,003 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-02 02:48:14,030 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on 01/26/2012
    20:08 GMT
    2012-12-02 02:48:14,030 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-02 02:48:14,030 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/34-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-02 02:48:14,031 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/34-zookeeper-server
    2012-12-02 02:48:14,039 INFO org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-02 02:48:14,039 INFO org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-02 02:48:14,040 INFO org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-02 02:48:14,072 INFO org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-02 02:48:14,088 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.28
    2012-12-02 02:48:14,109 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 2c
    2012-12-02 02:48:15,025 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:39458
    2012-12-02 02:48:15,029 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0002 at /
    10.40.222.77:39458
    2012-12-02 02:48:15,035 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0002 with negotiated timeout 40000 for
    client /10.40.222.77:39458
    2012-12-02 02:48:15,060 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:56157
    2012-12-02 02:48:15,062 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0000 at /
    10.83.35.173:56157
    2012-12-02 02:48:15,063 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0000 with negotiated timeout 40000 for
    client /10.83.35.173:56157
    2012-12-02 02:48:15,095 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:39459
    2012-12-02 02:48:15,095 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0001 at /
    10.40.222.77:39459
    2012-12-02 02:48:15,096 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0001 with negotiated timeout 40000 for
    client /10.40.222.77:39459
    2012-12-02 03:23:48,488 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.2d
    2012-12-02 03:23:48,560 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0001
    2012-12-02 03:23:48,570 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:39459 which had sessionid
    0x13b59e867fc0001
    2012-12-02 03:25:46,004 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54443
    2012-12-02 03:25:46,007 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54443
    2012-12-02 03:25:46,009 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30000 with negotiated timeout 40000 for
    client /10.40.222.77:54443
    2012-12-02 03:25:46,348 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30000, likely client has closed socket
    2012-12-02 03:25:46,349 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:54443 which had sessionid
    0x13b5a963ea30000
    2012-12-02 03:26:15,044 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b59e867fc0000, likely client has closed socket
    2012-12-02 03:26:15,045 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.83.35.173:56157 which had sessionid
    0x13b59e867fc0000
    2012-12-02 03:26:17,867 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b59e867fc0002, likely client has closed socket
    2012-12-02 03:26:17,867 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:39458 which had sessionid
    0x13b59e867fc0002
    2012-12-02 03:26:28,001 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30000, timeout of 40000ms exceeded
    2012-12-02 03:26:28,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30000
    2012-12-02 03:26:42,001 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b59e867fc0000, timeout of 40000ms exceeded
    2012-12-02 03:26:42,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0000
    2012-12-02 03:26:47,391 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54461
    2012-12-02 03:26:47,394 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54461
    2012-12-02 03:26:47,396 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30001 with negotiated timeout 40000 for
    client /10.40.222.77:54461
    2012-12-02 03:26:47,448 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54462
    2012-12-02 03:26:47,460 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54462
    2012-12-02 03:26:47,463 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30002 with negotiated timeout 40000 for
    client /10.40.222.77:54462
    2012-12-02 03:26:56,000 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b59e867fc0002, timeout of 40000ms exceeded
    2012-12-02 03:26:56,000 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0002
    2012-12-02 03:27:00,606 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:56794
    2012-12-02 03:27:00,614 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:56794
    2012-12-02 03:27:00,617 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30003 with negotiated timeout 40000 for
    client /10.83.35.173:56794
    2012-12-02 03:36:45,885 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/43-zookeeper-server/zoo.cfg
    2012-12-02 03:36:45,889 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-02 03:36:45,962 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/43-zookeeper-server/zoo.cfg
    2012-12-02 03:36:45,962 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on 01/26/2012
    20:08 GMT
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/43-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-02 03:36:45,976 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/43-zookeeper-server
    2012-12-02 03:36:45,985 INFO org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-02 03:36:45,985 INFO org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-02 03:36:45,985 INFO org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-02 03:36:46,024 INFO org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-02 03:36:46,042 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.2c
    2012-12-02 03:36:46,063 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 36
    2012-12-02 03:36:46,531 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:37153
    2012-12-02 03:36:46,535 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30003 at /
    10.83.35.173:37153
    2012-12-02 03:36:46,539 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30003 with negotiated timeout 40000 for
    client /10.83.35.173:37153
    2012-12-02 03:36:46,883 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:47487
    2012-12-02 03:36:46,884 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30002 at /
    10.40.222.77:47487
    2012-12-02 03:36:46,884 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30002 with negotiated timeout 40000 for
    client /10.40.222.77:47487
    2012-12-02 03:36:47,310 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:47488
    2012-12-02 03:36:47,311 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30001 at /
    10.40.222.77:47488
    2012-12-02 03:36:47,312 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30001 with negotiated timeout 40000 for
    client /10.40.222.77:47488
    2012-12-02 03:41:46,616 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44776
    2012-12-02 03:41:46,619 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44776
    2012-12-02 03:41:46,620 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.37
    2012-12-02 03:41:46,674 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0000 with negotiated timeout 40000 for
    client /10.40.222.77:44776
    2012-12-02 03:41:47,494 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5ac2ad7a0000, likely client has closed socket
    2012-12-02 03:41:47,495 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:44776 which had sessionid
    0x13b5ac2ad7a0000
    2012-12-02 03:42:16,001 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30003, likely client has closed socket
    2012-12-02 03:42:16,002 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.83.35.173:37153 which had sessionid
    0x13b5a963ea30003
    2012-12-02 03:42:17,916 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30002, likely client has closed socket
    2012-12-02 03:42:17,917 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:47487 which had sessionid
    0x13b5a963ea30002
    2012-12-02 03:42:19,468 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30001, likely client has closed socket
    2012-12-02 03:42:19,469 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:47488 which had sessionid
    0x13b5a963ea30001
    2012-12-02 03:42:28,000 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5ac2ad7a0000, timeout of 40000ms exceeded
    2012-12-02 03:42:28,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5ac2ad7a0000
    2012-12-02 03:42:46,214 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:52486
    2012-12-02 03:42:46,217 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:52486
    2012-12-02 03:42:46,220 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0001 with negotiated timeout 40000 for
    client /10.83.35.173:52486
    2012-12-02 03:42:47,857 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44799
    2012-12-02 03:42:47,864 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44799
    2012-12-02 03:42:47,867 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0002 with negotiated timeout 40000 for
    client /10.40.222.77:44799

    2012-12-02 03:42:47,991 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
    KeeperException when processing sessionid:0x13b5ac2ad7a0002 type:create
    cxid:0x6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
    Path:/hbase/master Error:KeeperErrorCode = NodeExists for /hbase/master
    2012-12-02 03:42:48,001 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30002, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30003, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30001, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30002
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30003
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30001
    2012-12-02 03:42:48,182 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44800
    2012-12-02 03:42:48,194 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44800
    2012-12-02 03:42:48,196 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0003 with negotiated timeout 40000 for
    client /10.40.222.77:44800
    Shouvanik,

    The errors look very much like
    http://zookeeper-user.578899.n2.nabble.com/Unusual-exception-td5632833.html

    Is your client on 10.40.222.77? It is connecting to ZK. Do you have any
    unusual network/firewall setting?

    Cheers,
    bc

    Regards,
    Shouvanik


    On Mon, Dec 3, 2012 at 1:44 AM, Harsh J wrote:

    Hi Shouvanik,

    Those logs are all INFO level and there's no issue in it. Can you
    share the issue/message you get when you mean "I am not able to start
    hbase" instead?

    On Mon, Dec 3, 2012 at 1:35 AM, Shouvanik Haldar
    wrote:
    Hi Harsh,

    I am not able to start hbase although hbase service is up. I am
    getting
    weird zookeeper exception. can you please help

    [root@ip-10-40-222-77 ~]# tail -f
    /var/log/zookeeper/zookeeper-cmf-zookeeper1-SERVER-ip-10-40-222-77.ec2.internal.log
    2012-12-02 03:42:47,991 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
    KeeperException when processing sessionid:0x13b5ac2ad7a0002
    type:create
    cxid:0x6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
    Path:/hbase/master Error:KeeperErrorCode = NodeExists for
    /hbase/master
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30002, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30003, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30001, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30002
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30003
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30001
    2012-12-02 03:42:48,182 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44800
    2012-12-02 03:42:48,194 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44800
    2012-12-02 03:42:48,196 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0003 with negotiated timeout 40000
    for
    client /10.40.222.77:44800


    Regards,
    Shouvanik Haldar


    On Mon, Dec 3, 2012 at 1:24 AM, Harsh J wrote:

    I replied to this over your duplicate post at

    https://groups.google.com/a/cloudera.org/d/topic/cdh-user/ft02DH52BZ4/discussion
    On Mon, Dec 3, 2012 at 12:59 AM, John H wrote:

    Hi,

    I installed Cloudera Manager(4.1.1) and CDH4 (4.1.2) using
    tarball
    on
    Ubuntu 10.04.4 (VM using virtual box). I was able to setup
    Cloudera
    manager
    server and agent, setup roles (NN,DN and Secondary NN) in pseudo
    distributed
    mode. I deployed the configuration to hosts and formatted name node
    successfully.

    When I started the services from Cloudera manager, I got message
    saying
    all
    3 (NN,DN,SNN) were started successfully, but I cannot access anyone
    of
    them
    through web url.
    The status for all 3 is started with good health.


    I looked at the log file for namenode and it is stuck at this line



    "2012-12-02 10:33:33,841 INFO
    org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager:
    Going
    to
    retain 2 images with txid >= 0"

    This is the first start after format, so I am confused why this is
    happening... Any help would be really appreciated..


    Thanks
    JH





    FULL LOG FILE: ***************************



    ************************************************************/
    2012-12-02 10:32:19,931 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting NameNode
    STARTUP_MSG: host = ubuntu1/127.0.0.1
    STARTUP_MSG: args = []
    STARTUP_MSG: version = 2.0.0-cdh4.1.2
    STARTUP_MSG: classpath =

    /home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-log4j12-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-json-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jettison-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jets3t-0.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-digester-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-annotations-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/activation-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/kfs-0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-net-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-auth-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsch-0.1.42.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-api-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/paranamer-2.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-math-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/junit-4.8.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/avro-1.7.1.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/stax-api-1.0.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-tests.jar:/home/john/cloudera-manager/cm-4.1.1/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-daemon-1.0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-tests.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-sources.jar:/usr/lib/hadoop-yarn/share/hadoop/yarn/*:/usr/lib/hadoop-mapreduce/share/hadoop/mapreduce/*
    STARTUP_MSG: build =

    file:///var/lib/jenkins/workspace/CDH4.1.2-Packaging-Hadoop/build/cdh4/hadoop/2.0.0-cdh4.1.2/source/hadoop-common-project/hadoop-common
    -r f0b53c81cbf56f5955e403b49fcd27afd5f082de; compiled by 'jenkins'
    on
    Thu
    Nov 1 17:06:55 PDT 2012
    ************************************************************/
    2012-12-02 10:32:22,214 INFO
    org.apache.hadoop.metrics2.impl.MetricsConfig:
    loaded properties from hadoop-metrics2.properties
    2012-12-02 10:32:23,240 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled
    snapshot
    period
    at 10 second(s).
    2012-12-02 10:32:23,240 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
    system
    started
    2012-12-02 10:32:25,269 INFO
    org.apache.hadoop.util.HostsFileReader:
    Refreshing hosts (include/exclude) list
    2012-12-02 10:32:25,270 INFO
    org.apache.hadoop.util.HostsFileReader:
    Adding
    127.0.0.1 to the list of hosts from

    /home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE/dfs_hosts_allow.txt
    2012-12-02 10:32:25,362 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
    dfs.block.invalidate.limit=1000
    2012-12-02 10:32:25,422 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    dfs.block.access.token.enable=false
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    defaultReplication = 1
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    maxReplication
    = 512
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    minReplication
    = 1
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    maxReplicationStreams = 2
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    shouldCheckForEnoughRacks = true
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    replicationRecheckInterval = 3000
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    encryptDataTransfer = false
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
    =
    john (auth:SIMPLE)
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
    =
    john
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    isPermissionEnabled
    =
    true
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled:
    false
    2012-12-02 10:32:25,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append
    Enabled:
    true
    2012-12-02 10:32:27,741 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
    occuring
    more than 10 times
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.threshold-pct = 0.9990000128746033
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.min.datanodes = 0
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.extension = 30000
    2012-12-02 10:32:27,941 INFO
    org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /home/john/dfs_dir/namenode/data1/in_use.lock acquired by
    nodename
    2070@ubuntu1
    2012-12-02 10:32:28,574 INFO
    org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /home/john/dfs_dir/namenode/data2/in_use.lock acquired by
    nodename
    2070@ubuntu1
    2012-12-02 10:32:28,803 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Recovering
    unfinalized segments in /home/john/dfs_dir/namenode/data1/current
    2012-12-02 10:32:28,803 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Recovering
    unfinalized segments in /home/john/dfs_dir/namenode/data2/current
    2012-12-02 10:32:28,805 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    No edit log streams selected.
    2012-12-02 10:32:28,950 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Loading image file
    /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000
    using
    no compression
    2012-12-02 10:32:28,950 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Number of files = 1
    2012-12-02 10:32:28,951 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Number of files under construction = 0
    2012-12-02 10:32:28,952 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Image file of size 117 loaded in 0 seconds.
    2012-12-02 10:32:28,952 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Loaded image for txid 0 from
    /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000
    2012-12-02 10:32:28,955 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
    segment
    at 1
    2012-12-02 10:32:30,041 INFO
    org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with
    entries
    0 lookups
    2012-12-02 10:32:30,041 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished
    loading
    FSImage in 2298 msecs
    2012-12-02 10:32:31,360 INFO org.apache.hadoop.ipc.Server: Starting
    Socket
    Reader #1 for port 8020
    2012-12-02 10:32:31,402 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
    FSNamesystemState MBean
    2012-12-02 10:32:31,482 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
    blocks
    under
    construction: 0
    2012-12-02 10:32:31,482 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: initializing
    replication queues
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Total
    number
    of
    blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number
    of
    invalid blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number
    of
    under-replicated blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number
    of
    over-replicated blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number
    of
    blocks
    being written = 0
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    Replication Queue initialization scan for invalid, over- and
    under-replicated blocks completed in 31 msec
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    Leaving safe mode after 6 secs.
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    Network topology has 0 racks and 0 datanodes
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    UnderReplicatedBlocks has 0 blocks
    2012-12-02 10:32:31,674 INFO org.mortbay.log: Logging to
    org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
    org.mortbay.log.Slf4jLog
    2012-12-02 10:32:31,814 INFO org.apache.hadoop.http.HttpServer:
    Added
    global
    filter 'safety'
    (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer:
    Added
    filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context hdfs
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer:
    Added
    filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context logs
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer:
    Added
    filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context static
    2012-12-02 10:32:31,841 INFO org.apache.hadoop.http.HttpServer:
    dfs.webhdfs.enabled = true
    2012-12-02 10:32:31,842 INFO org.apache.hadoop.http.HttpServer:
    Added
    filter
    'SPNEGO' (class=org.apache.hadoop.hdfs.web.AuthFilter)
    2012-12-02 10:32:31,844 INFO org.apache.hadoop.http.HttpServer:
    addJerseyResourcePackage:

    packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
    pathSpec=/webhdfs/v1/*
    2012-12-02 10:32:31,862 INFO org.apache.hadoop.http.HttpServer:
    Jetty
    bound
    to port 50070
    2012-12-02 10:32:31,862 INFO org.mortbay.log:
    jetty-6.1.26.cloudera.2
    2012-12-02 10:32:32,261 WARN
    org.apache.hadoop.security.authentication.server.AuthenticationFilter:
    'signature.secret' configuration not set, using a random value as
    secret
    2012-12-02 10:32:32,453 INFO org.mortbay.log: Started
    selectchannelconnector@localhost.localdomain:50070
    2012-12-02 10:32:32,453 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at:
    localhost.localdomain:50070
    2012-12-02 10:32:32,453 INFO org.apache.hadoop.ipc.Server: IPC
    Server
    Responder: starting
    2012-12-02 10:32:32,454 INFO org.apache.hadoop.ipc.Server: IPC
    Server
    listener on 8020: starting
    2012-12-02 10:32:32,603 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode up at:
    ubuntu1/127.0.0.1:8020
    2012-12-02 10:32:32,603 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting
    services
    required for active state
    2012-12-02 10:32:34,359 INFO
    org.apache.hadoop.fs.TrashPolicyDefault:
    The
    configured checkpoint interval is 0 minutes. Using an interval of
    1440
    minutes that is used for deletion instead
    2012-12-02 10:32:35,780 INFO org.apache.hadoop.hdfs.StateChange:
    BLOCK*
    NameSystem.registerDatanode: node registration from
    DatanodeRegistration(127.0.0.1,
    storageID=DS-377461142-127.0.0.1-50010-1354473155123,
    infoPort=50075,
    ipcPort=50020, storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0)
    storage
    DS-377461142-127.0.0.1-50010-1354473155123
    2012-12-02 10:32:35,790 INFO org.apache.hadoop.net.NetworkTopology:
    Adding a
    new node: /default/127.0.0.1:50010
    2012-12-02 10:32:35,951 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: BLOCK*
    processReport: Received first block report from 127.0.0.1:50010after
    becoming active. Its block contents are no longer considered stale.
    2012-12-02 10:32:35,951 INFO org.apache.hadoop.hdfs.StateChange:
    BLOCK*
    processReport: from DatanodeRegistration(127.0.0.1,
    storageID=DS-377461142-127.0.0.1-50010-1354473155123,
    infoPort=50075,
    ipcPort=50020, storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0),
    blocks:
    0, processing time: 1 msecs
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
    from
    127.0.0.1
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit
    logs.
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
    segment
    1
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
    transactions: 2
    Total time for transactions(ms): 1Number of transactions batched in
    Syncs: 0
    Number of syncs: 2 SyncTimes(ms): 166 189
    2012-12-02 10:33:31,595 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
    transactions: 2
    Total time for transactions(ms): 1Number of transactions batched in
    Syncs: 0
    Number of syncs: 3 SyncTimes(ms): 215 255
    2012-12-02 10:33:31,603 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Finalizing
    edits
    file

    /home/john/dfs_dir/namenode/data1/current/edits_inprogress_0000000000000000001
    ->

    /home/john/dfs_dir/namenode/data1/current/edits_0000000000000000001-0000000000000000002
    2012-12-02 10:33:31,604 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Finalizing
    edits
    file

    /home/john/dfs_dir/namenode/data2/current/edits_inprogress_0000000000000000001
    ->

    /home/john/dfs_dir/namenode/data2/current/edits_0000000000000000001-0000000000000000002
    2012-12-02 10:33:31,604 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
    segment
    at 3
    2012-12-02 10:33:33,274 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Opening
    connection
    to

    http://ubuntu1:50090/getimage?getimage=1&txid=2&storageInfo=-40:829593698:0:cluster2
    2012-12-02 10:33:33,622 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Transfer
    took
    0.34s
    at 0.00 KB/s
    2012-12-02 10:33:33,622 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Downloaded
    file
    fsimage.ckpt_0000000000000000002 size 117 bytes.
    2012-12-02 10:33:33,841 INFO
    org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager:
    Going
    to
    retain 2 images with txid >= 0



    --
    Harsh J



    --
    Thanks,
    Shouvanik


    --
    Harsh J



    --
    Thanks,
    Shouvanik


    --
    Harsh J
  • Shouvanik Haldar at Dec 3, 2012 at 7:18 am
    Hi,

    Cloudera and all the services require some ports. These are enabled in AWS
    security group. Except that I am not aware of any firewall settings.

    Tell me one more thing. Does the /etc/hosts have anything to do?

    Let me paste the contents of this file.

    *127.0.0.1 localhost
    10.40.222.77 ip-10-40-222-77*


    and for the other server

    *127.0.0.1 localhost
    10.83.35.173 ip-10-83-35-173*

    Is this OK?
    Another help.
    Can you please point to the exact guide to upgrade to CM 4.x?

    Regards,
    Shouvanik


    On Mon, Dec 3, 2012 at 12:15 PM, bc Wong wrote:
    On Sun, Dec 2, 2012 at 12:34 PM, Harsh J wrote:

    Hi Shouvanik,

    I notice you use CDH3. You should really be using CDH4, as it provides
    a great deal of improvements over the CDH3 version that you'll be
    missing out on.

    In any case:

    If there exists a /etc/zookeeper/zoo.cfg on your client machine,
    please rename it to something else, such as zoo.cfg.unwanted, or
    remove it altogether, and retry.

    Alternatively, try to run your HBase shell command(s) with the direct
    script (instead of wrapper /usr/bin/hbase) as:

    /usr/lib/hbase/bin/hbase shell

    And see if it lets you connect and work normally this time.

    On Mon, Dec 3, 2012 at 1:50 AM, Shouvanik Haldar
    wrote:
    Yes Harsh,

    [root@ip-10-40-222-77 ~]# hbase shell
    HBase Shell; enter 'help<RETURN>' for list of supported commands.
    Type "exit<RETURN>" to leave the HBase Shell
    Version 0.90.4-cdh3u3, r, Thu Jan 26 10:13:36 PST 2012

    hbase(main):001:0> describe 't1'

    ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able
    to connect to ZooKeeper but the connection closes immediately. This could be
    a sign that the server has too many connections (30 is the default).
    Consider inspecting your ZK server logs for that error and then make sure
    you are reusing HBaseConfiguration as often as you can. See HTable's javadoc
    for more information.

    Here is some help for this command:
    Describe the named table. For example:
    hbase> describe 't1'


    hbase(main):002:0>


    Secondly,

    When I see the zookeeper logs, it says

    [root@ip-10-40-222-77 ~]# cat
    /var/log/zookeeper/zookeeper-cmf-zookeeper1-SERVER-ip-10-40-222-77.ec2.internal.log
    2012-12-01 23:30:35,726 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/1-zookeeper-server/zoo.cfg
    2012-12-01 23:30:35,731 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-01 23:30:35,811 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/1-zookeeper-server/zoo.cfg
    2012-12-01 23:30:35,812 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-01 23:30:35,842 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/1-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/1-zookeeper-server
    2012-12-01 23:30:35,851 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-01 23:30:35,851 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-01 23:30:35,851 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-01 23:30:35,891 INFO org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-01 23:30:35,907 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.0
    2012-12-01 23:30:35,941 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 28
    2012-12-01 23:38:21,260 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/9-zookeeper-server/zoo.cfg
    2012-12-01 23:38:21,265 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-01 23:38:21,339 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/9-zookeeper-server/zoo.cfg
    2012-12-01 23:38:21,340 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/9-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/9-zookeeper-server
    2012-12-01 23:38:21,366 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-01 23:38:21,366 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-01 23:38:21,366 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-01 23:38:21,399 INFO org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-01 23:38:21,415 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.28
    2012-12-01 23:38:21,430 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 28
    2012-12-01 23:43:49,121 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:45741
    2012-12-01 23:43:49,128 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:45741
    2012-12-01 23:43:49,130 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.29
    2012-12-01 23:43:49,172 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0000 with negotiated timeout 40000 for
    client /10.83.35.173:45741
    2012-12-01 23:43:52,894 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:59824
    2012-12-01 23:43:52,897 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:59824
    2012-12-01 23:43:52,899 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0001 with negotiated timeout 40000 for
    client /10.40.222.77:59824
    2012-12-01 23:43:52,993 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:59825
    2012-12-01 23:43:52,996 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:59825
    2012-12-01 23:43:52,998 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0002 with negotiated timeout 40000 for
    client /10.40.222.77:59825
    2012-12-02 02:48:13,910 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/34-zookeeper-server/zoo.cfg
    2012-12-02 02:48:13,914 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-02 02:48:14,002 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/34-zookeeper-server/zoo.cfg
    2012-12-02 02:48:14,003 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-02 02:48:14,030 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-02 02:48:14,030 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-02 02:48:14,030 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/34-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/34-zookeeper-server
    2012-12-02 02:48:14,039 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-02 02:48:14,039 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-02 02:48:14,040 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-02 02:48:14,072 INFO org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-02 02:48:14,088 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.28
    2012-12-02 02:48:14,109 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 2c
    2012-12-02 02:48:15,025 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:39458
    2012-12-02 02:48:15,029 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0002 at /
    10.40.222.77:39458
    2012-12-02 02:48:15,035 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0002 with negotiated timeout 40000 for
    client /10.40.222.77:39458
    2012-12-02 02:48:15,060 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:56157
    2012-12-02 02:48:15,062 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0000 at /
    10.83.35.173:56157
    2012-12-02 02:48:15,063 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0000 with negotiated timeout 40000 for
    client /10.83.35.173:56157
    2012-12-02 02:48:15,095 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:39459
    2012-12-02 02:48:15,095 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0001 at /
    10.40.222.77:39459
    2012-12-02 02:48:15,096 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0001 with negotiated timeout 40000 for
    client /10.40.222.77:39459
    2012-12-02 03:23:48,488 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.2d
    2012-12-02 03:23:48,560 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0001
    2012-12-02 03:23:48,570 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:39459 which had sessionid
    0x13b59e867fc0001
    2012-12-02 03:25:46,004 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54443
    2012-12-02 03:25:46,007 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54443
    2012-12-02 03:25:46,009 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30000 with negotiated timeout 40000 for
    client /10.40.222.77:54443
    2012-12-02 03:25:46,348 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30000, likely client has closed socket
    2012-12-02 03:25:46,349 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:54443 which had sessionid
    0x13b5a963ea30000
    2012-12-02 03:26:15,044 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b59e867fc0000, likely client has closed socket
    2012-12-02 03:26:15,045 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.83.35.173:56157 which had sessionid
    0x13b59e867fc0000
    2012-12-02 03:26:17,867 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b59e867fc0002, likely client has closed socket
    2012-12-02 03:26:17,867 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:39458 which had sessionid
    0x13b59e867fc0002
    2012-12-02 03:26:28,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30000, timeout of 40000ms exceeded
    2012-12-02 03:26:28,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30000
    2012-12-02 03:26:42,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b59e867fc0000, timeout of 40000ms exceeded
    2012-12-02 03:26:42,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0000
    2012-12-02 03:26:47,391 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54461
    2012-12-02 03:26:47,394 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54461
    2012-12-02 03:26:47,396 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30001 with negotiated timeout 40000 for
    client /10.40.222.77:54461
    2012-12-02 03:26:47,448 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54462
    2012-12-02 03:26:47,460 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54462
    2012-12-02 03:26:47,463 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30002 with negotiated timeout 40000 for
    client /10.40.222.77:54462
    2012-12-02 03:26:56,000 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b59e867fc0002, timeout of 40000ms exceeded
    2012-12-02 03:26:56,000 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0002
    2012-12-02 03:27:00,606 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:56794
    2012-12-02 03:27:00,614 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:56794
    2012-12-02 03:27:00,617 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30003 with negotiated timeout 40000 for
    client /10.83.35.173:56794
    2012-12-02 03:36:45,885 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/43-zookeeper-server/zoo.cfg
    2012-12-02 03:36:45,889 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-02 03:36:45,962 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/43-zookeeper-server/zoo.cfg
    2012-12-02 03:36:45,962 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/43-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/43-zookeeper-server
    2012-12-02 03:36:45,985 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-02 03:36:45,985 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-02 03:36:45,985 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-02 03:36:46,024 INFO org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-02 03:36:46,042 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.2c
    2012-12-02 03:36:46,063 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 36
    2012-12-02 03:36:46,531 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:37153
    2012-12-02 03:36:46,535 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30003 at /
    10.83.35.173:37153
    2012-12-02 03:36:46,539 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30003 with negotiated timeout 40000 for
    client /10.83.35.173:37153
    2012-12-02 03:36:46,883 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:47487
    2012-12-02 03:36:46,884 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30002 at /
    10.40.222.77:47487
    2012-12-02 03:36:46,884 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30002 with negotiated timeout 40000 for
    client /10.40.222.77:47487
    2012-12-02 03:36:47,310 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:47488
    2012-12-02 03:36:47,311 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30001 at /
    10.40.222.77:47488
    2012-12-02 03:36:47,312 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30001 with negotiated timeout 40000 for
    client /10.40.222.77:47488
    2012-12-02 03:41:46,616 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44776
    2012-12-02 03:41:46,619 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44776
    2012-12-02 03:41:46,620 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.37
    2012-12-02 03:41:46,674 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0000 with negotiated timeout 40000 for
    client /10.40.222.77:44776
    2012-12-02 03:41:47,494 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5ac2ad7a0000, likely client has closed socket
    2012-12-02 03:41:47,495 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:44776 which had sessionid
    0x13b5ac2ad7a0000
    2012-12-02 03:42:16,001 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30003, likely client has closed socket
    2012-12-02 03:42:16,002 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.83.35.173:37153 which had sessionid
    0x13b5a963ea30003
    2012-12-02 03:42:17,916 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30002, likely client has closed socket
    2012-12-02 03:42:17,917 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:47487 which had sessionid
    0x13b5a963ea30002
    2012-12-02 03:42:19,468 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30001, likely client has closed socket
    2012-12-02 03:42:19,469 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:47488 which had sessionid
    0x13b5a963ea30001
    2012-12-02 03:42:28,000 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5ac2ad7a0000, timeout of 40000ms exceeded
    2012-12-02 03:42:28,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5ac2ad7a0000
    2012-12-02 03:42:46,214 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:52486
    2012-12-02 03:42:46,217 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:52486
    2012-12-02 03:42:46,220 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0001 with negotiated timeout 40000 for
    client /10.83.35.173:52486
    2012-12-02 03:42:47,857 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44799
    2012-12-02 03:42:47,864 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44799
    2012-12-02 03:42:47,867 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0002 with negotiated timeout 40000 for
    client /10.40.222.77:44799

    2012-12-02 03:42:47,991 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
    KeeperException when processing sessionid:0x13b5ac2ad7a0002 type:create
    cxid:0x6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
    Path:/hbase/master Error:KeeperErrorCode = NodeExists for /hbase/master
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30002, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30003, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30001, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30002
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30003
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30001
    2012-12-02 03:42:48,182 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44800
    2012-12-02 03:42:48,194 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44800
    2012-12-02 03:42:48,196 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0003 with negotiated timeout 40000 for
    client /10.40.222.77:44800
    Shouvanik,

    The errors look very much like
    http://zookeeper-user.578899.n2.nabble.com/Unusual-exception-td5632833.html

    Is your client on 10.40.222.77? It is connecting to ZK. Do you have any
    unusual network/firewall setting?

    Cheers,
    bc

    Regards,
    Shouvanik


    On Mon, Dec 3, 2012 at 1:44 AM, Harsh J wrote:

    Hi Shouvanik,

    Those logs are all INFO level and there's no issue in it. Can you
    share the issue/message you get when you mean "I am not able to start
    hbase" instead?

    On Mon, Dec 3, 2012 at 1:35 AM, Shouvanik Haldar
    wrote:
    Hi Harsh,

    I am not able to start hbase although hbase service is up. I am
    getting
    weird zookeeper exception. can you please help

    [root@ip-10-40-222-77 ~]# tail -f
    /var/log/zookeeper/zookeeper-cmf-zookeeper1-SERVER-ip-10-40-222-77.ec2.internal.log
    2012-12-02 03:42:47,991 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
    KeeperException when processing sessionid:0x13b5ac2ad7a0002
    type:create
    cxid:0x6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
    Path:/hbase/master Error:KeeperErrorCode = NodeExists for
    /hbase/master
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30002, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30003, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30001, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30002
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30003
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30001
    2012-12-02 03:42:48,182 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44800
    2012-12-02 03:42:48,194 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44800
    2012-12-02 03:42:48,196 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0003 with negotiated timeout 40000
    for
    client /10.40.222.77:44800


    Regards,
    Shouvanik Haldar


    On Mon, Dec 3, 2012 at 1:24 AM, Harsh J wrote:

    I replied to this over your duplicate post at

    https://groups.google.com/a/cloudera.org/d/topic/cdh-user/ft02DH52BZ4/discussion
    On Mon, Dec 3, 2012 at 12:59 AM, John H <john.hadoop@gmail.com>
    wrote:
    Hi,

    I installed Cloudera Manager(4.1.1) and CDH4 (4.1.2) using
    tarball
    on
    Ubuntu 10.04.4 (VM using virtual box). I was able to setup
    Cloudera
    manager
    server and agent, setup roles (NN,DN and Secondary NN) in pseudo
    distributed
    mode. I deployed the configuration to hosts and formatted name
    node
    successfully.

    When I started the services from Cloudera manager, I got message
    saying
    all
    3 (NN,DN,SNN) were started successfully, but I cannot access
    anyone
    of
    them
    through web url.
    The status for all 3 is started with good health.


    I looked at the log file for namenode and it is stuck at this line



    "2012-12-02 10:33:33,841 INFO
    org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager:
    Going
    to
    retain 2 images with txid >= 0"

    This is the first start after format, so I am confused why this is
    happening... Any help would be really appreciated..


    Thanks
    JH





    FULL LOG FILE: ***************************



    ************************************************************/
    2012-12-02 10:32:19,931 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting NameNode
    STARTUP_MSG: host = ubuntu1/127.0.0.1
    STARTUP_MSG: args = []
    STARTUP_MSG: version = 2.0.0-cdh4.1.2
    STARTUP_MSG: classpath =

    /home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-log4j12-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-json-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jettison-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jets3t-0.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-digester-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-annotations-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/activation-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/kfs-0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-net-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-auth-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsch-0.1.42.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-api-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/paranamer-2.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-math-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/junit-4.8.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/avro-1.7.1.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/stax-api-1.0.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-tests.jar:/home/john/cloudera-manager/cm-4.1.1/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-daemon-1.0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-tests.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-sources.jar:/usr/lib/hadoop-yarn/share/hadoop/yarn/*:/usr/lib/hadoop-mapreduce/share/hadoop/mapreduce/*
    STARTUP_MSG: build =

    file:///var/lib/jenkins/workspace/CDH4.1.2-Packaging-Hadoop/build/cdh4/hadoop/2.0.0-cdh4.1.2/source/hadoop-common-project/hadoop-common
    -r f0b53c81cbf56f5955e403b49fcd27afd5f082de; compiled by
    'jenkins' on
    Thu
    Nov 1 17:06:55 PDT 2012
    ************************************************************/
    2012-12-02 10:32:22,214 INFO
    org.apache.hadoop.metrics2.impl.MetricsConfig:
    loaded properties from hadoop-metrics2.properties
    2012-12-02 10:32:23,240 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled
    snapshot
    period
    at 10 second(s).
    2012-12-02 10:32:23,240 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode
    metrics
    system
    started
    2012-12-02 10:32:25,269 INFO
    org.apache.hadoop.util.HostsFileReader:
    Refreshing hosts (include/exclude) list
    2012-12-02 10:32:25,270 INFO
    org.apache.hadoop.util.HostsFileReader:
    Adding
    127.0.0.1 to the list of hosts from

    /home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE/dfs_hosts_allow.txt
    2012-12-02 10:32:25,362 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
    dfs.block.invalidate.limit=1000
    2012-12-02 10:32:25,422 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    dfs.block.access.token.enable=false
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    defaultReplication = 1
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    maxReplication
    = 512
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    minReplication
    = 1
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    maxReplicationStreams = 2
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    shouldCheckForEnoughRacks = true
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    replicationRecheckInterval = 3000
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    encryptDataTransfer = false
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
    =
    john (auth:SIMPLE)
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
    =
    john
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    isPermissionEnabled
    =
    true
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled:
    false
    2012-12-02 10:32:25,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append
    Enabled:
    true
    2012-12-02 10:32:27,741 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file
    names
    occuring
    more than 10 times
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.threshold-pct = 0.9990000128746033
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.min.datanodes = 0
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.extension = 30000
    2012-12-02 10:32:27,941 INFO
    org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /home/john/dfs_dir/namenode/data1/in_use.lock acquired by
    nodename
    2070@ubuntu1
    2012-12-02 10:32:28,574 INFO
    org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /home/john/dfs_dir/namenode/data2/in_use.lock acquired by
    nodename
    2070@ubuntu1
    2012-12-02 10:32:28,803 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Recovering
    unfinalized segments in /home/john/dfs_dir/namenode/data1/current
    2012-12-02 10:32:28,803 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Recovering
    unfinalized segments in /home/john/dfs_dir/namenode/data2/current
    2012-12-02 10:32:28,805 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    No edit log streams selected.
    2012-12-02 10:32:28,950 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Loading image file
    /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000
    using
    no compression
    2012-12-02 10:32:28,950 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Number of files = 1
    2012-12-02 10:32:28,951 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Number of files under construction = 0
    2012-12-02 10:32:28,952 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Image file of size 117 loaded in 0 seconds.
    2012-12-02 10:32:28,952 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Loaded image for txid 0 from
    /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000
    2012-12-02 10:32:28,955 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
    segment
    at 1
    2012-12-02 10:32:30,041 INFO
    org.apache.hadoop.hdfs.server.namenode.NameCache: initialized
    with 0
    entries
    0 lookups
    2012-12-02 10:32:30,041 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished
    loading
    FSImage in 2298 msecs
    2012-12-02 10:32:31,360 INFO org.apache.hadoop.ipc.Server:
    Starting
    Socket
    Reader #1 for port 8020
    2012-12-02 10:32:31,402 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
    FSNamesystemState MBean
    2012-12-02 10:32:31,482 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
    blocks
    under
    construction: 0
    2012-12-02 10:32:31,482 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: initializing
    replication queues
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Total
    number
    of
    blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number of
    invalid blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number of
    under-replicated blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number of
    over-replicated blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number of
    blocks
    being written = 0
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    Replication Queue initialization scan for invalid, over- and
    under-replicated blocks completed in 31 msec
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    Leaving safe mode after 6 secs.
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    Network topology has 0 racks and 0 datanodes
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    UnderReplicatedBlocks has 0 blocks
    2012-12-02 10:32:31,674 INFO org.mortbay.log: Logging to
    org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
    org.mortbay.log.Slf4jLog
    2012-12-02 10:32:31,814 INFO org.apache.hadoop.http.HttpServer:
    Added
    global
    filter 'safety'
    (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer:
    Added
    filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context hdfs
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer:
    Added
    filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context logs
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer:
    Added
    filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context static
    2012-12-02 10:32:31,841 INFO org.apache.hadoop.http.HttpServer:
    dfs.webhdfs.enabled = true
    2012-12-02 10:32:31,842 INFO org.apache.hadoop.http.HttpServer:
    Added
    filter
    'SPNEGO' (class=org.apache.hadoop.hdfs.web.AuthFilter)
    2012-12-02 10:32:31,844 INFO org.apache.hadoop.http.HttpServer:
    addJerseyResourcePackage:

    packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
    pathSpec=/webhdfs/v1/*
    2012-12-02 10:32:31,862 INFO org.apache.hadoop.http.HttpServer:
    Jetty
    bound
    to port 50070
    2012-12-02 10:32:31,862 INFO org.mortbay.log:
    jetty-6.1.26.cloudera.2
    2012-12-02 10:32:32,261 WARN
    org.apache.hadoop.security.authentication.server.AuthenticationFilter:
    'signature.secret' configuration not set, using a random value as
    secret
    2012-12-02 10:32:32,453 INFO org.mortbay.log: Started
    selectchannelconnector@localhost.localdomain:50070
    2012-12-02 10:32:32,453 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at:
    localhost.localdomain:50070
    2012-12-02 10:32:32,453 INFO org.apache.hadoop.ipc.Server: IPC
    Server
    Responder: starting
    2012-12-02 10:32:32,454 INFO org.apache.hadoop.ipc.Server: IPC
    Server
    listener on 8020: starting
    2012-12-02 10:32:32,603 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode up at:
    ubuntu1/127.0.0.1:8020
    2012-12-02 10:32:32,603 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting
    services
    required for active state
    2012-12-02 10:32:34,359 INFO
    org.apache.hadoop.fs.TrashPolicyDefault:
    The
    configured checkpoint interval is 0 minutes. Using an interval of
    1440
    minutes that is used for deletion instead
    2012-12-02 10:32:35,780 INFO org.apache.hadoop.hdfs.StateChange:
    BLOCK*
    NameSystem.registerDatanode: node registration from
    DatanodeRegistration(127.0.0.1,
    storageID=DS-377461142-127.0.0.1-50010-1354473155123,
    infoPort=50075,
    ipcPort=50020, storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0)
    storage
    DS-377461142-127.0.0.1-50010-1354473155123
    2012-12-02 10:32:35,790 INFO
    org.apache.hadoop.net.NetworkTopology:
    Adding a
    new node: /default/127.0.0.1:50010
    2012-12-02 10:32:35,951 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: BLOCK*
    processReport: Received first block report from 127.0.0.1:50010after
    becoming active. Its block contents are no longer considered
    stale.
    2012-12-02 10:32:35,951 INFO org.apache.hadoop.hdfs.StateChange:
    BLOCK*
    processReport: from DatanodeRegistration(127.0.0.1,
    storageID=DS-377461142-127.0.0.1-50010-1354473155123,
    infoPort=50075,
    ipcPort=50020,
    storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0),
    blocks:
    0, processing time: 1 msecs
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
    from
    127.0.0.1
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit
    logs.
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
    segment
    1
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
    transactions: 2
    Total time for transactions(ms): 1Number of transactions batched
    in
    Syncs: 0
    Number of syncs: 2 SyncTimes(ms): 166 189
    2012-12-02 10:33:31,595 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
    transactions: 2
    Total time for transactions(ms): 1Number of transactions batched
    in
    Syncs: 0
    Number of syncs: 3 SyncTimes(ms): 215 255
    2012-12-02 10:33:31,603 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Finalizing
    edits
    file

    /home/john/dfs_dir/namenode/data1/current/edits_inprogress_0000000000000000001
    ->

    /home/john/dfs_dir/namenode/data1/current/edits_0000000000000000001-0000000000000000002
    2012-12-02 10:33:31,604 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Finalizing
    edits
    file

    /home/john/dfs_dir/namenode/data2/current/edits_inprogress_0000000000000000001
    ->

    /home/john/dfs_dir/namenode/data2/current/edits_0000000000000000001-0000000000000000002
    2012-12-02 10:33:31,604 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
    segment
    at 3
    2012-12-02 10:33:33,274 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Opening
    connection
    to

    http://ubuntu1:50090/getimage?getimage=1&txid=2&storageInfo=-40:829593698:0:cluster2
    2012-12-02 10:33:33,622 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Transfer
    took
    0.34s
    at 0.00 KB/s
    2012-12-02 10:33:33,622 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Downloaded
    file
    fsimage.ckpt_0000000000000000002 size 117 bytes.
    2012-12-02 10:33:33,841 INFO
    org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager:
    Going
    to
    retain 2 images with txid >= 0



    --
    Harsh J



    --
    Thanks,
    Shouvanik


    --
    Harsh J



    --
    Thanks,
    Shouvanik


    --
    Harsh J

    --
    Thanks,
    *Shouvanik*
  • bc Wong at Dec 3, 2012 at 8:12 am

    On Sun, Dec 2, 2012 at 11:18 PM, Shouvanik Haldar wrote:

    Hi,

    Cloudera and all the services require some ports. These are enabled in AWS
    security group. Except that I am not aware of any firewall settings.

    Tell me one more thing. Does the /etc/hosts have anything to do?

    Let me paste the contents of this file.

    *127.0.0.1 localhost
    10.40.222.77 ip-10-40-222-77*


    and for the other server

    *127.0.0.1 localhost
    10.83.35.173 ip-10-83-35-173*

    Is this OK?
    Looks fine. Maybe you should heed this warning:

    "ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able
    to connect to ZooKeeper but the connection closes immediately. This could be
    a sign that the server has too many connections (30 is the default).
    Consider inspecting your ZK server logs for that error and then make sure
    you are reusing HBaseConfiguration as often as you can. See HTable's javadoc
    for more information."

    Can you restart your ZK and HBase services, and them immediately try to do
    a "describe".

    Another help.
    Can you please point to the exact guide to upgrade to CM 4.x?
    https://ccp.cloudera.com/display/ENT4DOC/Upgrading+to+Cloudera+Manager+4.0

    See the subsections on upgrading from/to different versions.

    Cheers,
    bc

    Regards,
    Shouvanik



    On Mon, Dec 3, 2012 at 12:15 PM, bc Wong wrote:
    On Sun, Dec 2, 2012 at 12:34 PM, Harsh J wrote:

    Hi Shouvanik,

    I notice you use CDH3. You should really be using CDH4, as it provides
    a great deal of improvements over the CDH3 version that you'll be
    missing out on.

    In any case:

    If there exists a /etc/zookeeper/zoo.cfg on your client machine,
    please rename it to something else, such as zoo.cfg.unwanted, or
    remove it altogether, and retry.

    Alternatively, try to run your HBase shell command(s) with the direct
    script (instead of wrapper /usr/bin/hbase) as:

    /usr/lib/hbase/bin/hbase shell

    And see if it lets you connect and work normally this time.

    On Mon, Dec 3, 2012 at 1:50 AM, Shouvanik Haldar
    wrote:
    Yes Harsh,

    [root@ip-10-40-222-77 ~]# hbase shell
    HBase Shell; enter 'help<RETURN>' for list of supported commands.
    Type "exit<RETURN>" to leave the HBase Shell
    Version 0.90.4-cdh3u3, r, Thu Jan 26 10:13:36 PST 2012

    hbase(main):001:0> describe 't1'

    ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able
    to connect to ZooKeeper but the connection closes immediately. This could be
    a sign that the server has too many connections (30 is the default).
    Consider inspecting your ZK server logs for that error and then make sure
    you are reusing HBaseConfiguration as often as you can. See HTable's javadoc
    for more information.

    Here is some help for this command:
    Describe the named table. For example:
    hbase> describe 't1'


    hbase(main):002:0>


    Secondly,

    When I see the zookeeper logs, it says

    [root@ip-10-40-222-77 ~]# cat
    /var/log/zookeeper/zookeeper-cmf-zookeeper1-SERVER-ip-10-40-222-77.ec2.internal.log
    2012-12-01 23:30:35,726 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/1-zookeeper-server/zoo.cfg
    2012-12-01 23:30:35,731 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-01 23:30:35,811 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/1-zookeeper-server/zoo.cfg
    2012-12-01 23:30:35,812 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-01 23:30:35,842 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/1-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/1-zookeeper-server
    2012-12-01 23:30:35,851 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-01 23:30:35,851 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-01 23:30:35,851 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-01 23:30:35,891 INFO org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-01 23:30:35,907 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.0
    2012-12-01 23:30:35,941 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 28
    2012-12-01 23:38:21,260 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/9-zookeeper-server/zoo.cfg
    2012-12-01 23:38:21,265 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-01 23:38:21,339 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/9-zookeeper-server/zoo.cfg
    2012-12-01 23:38:21,340 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/9-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/9-zookeeper-server
    2012-12-01 23:38:21,366 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-01 23:38:21,366 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-01 23:38:21,366 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-01 23:38:21,399 INFO org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-01 23:38:21,415 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.28
    2012-12-01 23:38:21,430 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 28
    2012-12-01 23:43:49,121 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:45741
    2012-12-01 23:43:49,128 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:45741
    2012-12-01 23:43:49,130 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.29
    2012-12-01 23:43:49,172 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0000 with negotiated timeout 40000 for
    client /10.83.35.173:45741
    2012-12-01 23:43:52,894 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:59824
    2012-12-01 23:43:52,897 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:59824
    2012-12-01 23:43:52,899 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0001 with negotiated timeout 40000 for
    client /10.40.222.77:59824
    2012-12-01 23:43:52,993 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:59825
    2012-12-01 23:43:52,996 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:59825
    2012-12-01 23:43:52,998 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0002 with negotiated timeout 40000 for
    client /10.40.222.77:59825
    2012-12-02 02:48:13,910 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/34-zookeeper-server/zoo.cfg
    2012-12-02 02:48:13,914 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-02 02:48:14,002 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/34-zookeeper-server/zoo.cfg
    2012-12-02 02:48:14,003 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-02 02:48:14,030 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-02 02:48:14,030 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-02 02:48:14,030 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/34-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/34-zookeeper-server
    2012-12-02 02:48:14,039 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-02 02:48:14,039 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-02 02:48:14,040 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-02 02:48:14,072 INFO org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-02 02:48:14,088 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.28
    2012-12-02 02:48:14,109 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 2c
    2012-12-02 02:48:15,025 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:39458
    2012-12-02 02:48:15,029 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0002 at /
    10.40.222.77:39458
    2012-12-02 02:48:15,035 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0002 with negotiated timeout 40000 for
    client /10.40.222.77:39458
    2012-12-02 02:48:15,060 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:56157
    2012-12-02 02:48:15,062 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0000 at /
    10.83.35.173:56157
    2012-12-02 02:48:15,063 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0000 with negotiated timeout 40000 for
    client /10.83.35.173:56157
    2012-12-02 02:48:15,095 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:39459
    2012-12-02 02:48:15,095 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0001 at /
    10.40.222.77:39459
    2012-12-02 02:48:15,096 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0001 with negotiated timeout 40000 for
    client /10.40.222.77:39459
    2012-12-02 03:23:48,488 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.2d
    2012-12-02 03:23:48,560 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0001
    2012-12-02 03:23:48,570 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:39459 which had sessionid
    0x13b59e867fc0001
    2012-12-02 03:25:46,004 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54443
    2012-12-02 03:25:46,007 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54443
    2012-12-02 03:25:46,009 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30000 with negotiated timeout 40000 for
    client /10.40.222.77:54443
    2012-12-02 03:25:46,348 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30000, likely client has closed socket
    2012-12-02 03:25:46,349 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:54443 which had sessionid
    0x13b5a963ea30000
    2012-12-02 03:26:15,044 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b59e867fc0000, likely client has closed socket
    2012-12-02 03:26:15,045 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.83.35.173:56157 which had sessionid
    0x13b59e867fc0000
    2012-12-02 03:26:17,867 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b59e867fc0002, likely client has closed socket
    2012-12-02 03:26:17,867 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:39458 which had sessionid
    0x13b59e867fc0002
    2012-12-02 03:26:28,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30000, timeout of 40000ms exceeded
    2012-12-02 03:26:28,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30000
    2012-12-02 03:26:42,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b59e867fc0000, timeout of 40000ms exceeded
    2012-12-02 03:26:42,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0000
    2012-12-02 03:26:47,391 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54461
    2012-12-02 03:26:47,394 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54461
    2012-12-02 03:26:47,396 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30001 with negotiated timeout 40000 for
    client /10.40.222.77:54461
    2012-12-02 03:26:47,448 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54462
    2012-12-02 03:26:47,460 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54462
    2012-12-02 03:26:47,463 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30002 with negotiated timeout 40000 for
    client /10.40.222.77:54462
    2012-12-02 03:26:56,000 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b59e867fc0002, timeout of 40000ms exceeded
    2012-12-02 03:26:56,000 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0002
    2012-12-02 03:27:00,606 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:56794
    2012-12-02 03:27:00,614 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:56794
    2012-12-02 03:27:00,617 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30003 with negotiated timeout 40000 for
    client /10.83.35.173:56794
    2012-12-02 03:36:45,885 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/43-zookeeper-server/zoo.cfg
    2012-12-02 03:36:45,889 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-02 03:36:45,962 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/43-zookeeper-server/zoo.cfg
    2012-12-02 03:36:45,962 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/43-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/43-zookeeper-server
    2012-12-02 03:36:45,985 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-02 03:36:45,985 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-02 03:36:45,985 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-02 03:36:46,024 INFO org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-02 03:36:46,042 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.2c
    2012-12-02 03:36:46,063 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 36
    2012-12-02 03:36:46,531 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:37153
    2012-12-02 03:36:46,535 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30003 at /
    10.83.35.173:37153
    2012-12-02 03:36:46,539 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30003 with negotiated timeout 40000 for
    client /10.83.35.173:37153
    2012-12-02 03:36:46,883 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:47487
    2012-12-02 03:36:46,884 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30002 at /
    10.40.222.77:47487
    2012-12-02 03:36:46,884 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30002 with negotiated timeout 40000 for
    client /10.40.222.77:47487
    2012-12-02 03:36:47,310 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:47488
    2012-12-02 03:36:47,311 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30001 at /
    10.40.222.77:47488
    2012-12-02 03:36:47,312 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30001 with negotiated timeout 40000 for
    client /10.40.222.77:47488
    2012-12-02 03:41:46,616 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44776
    2012-12-02 03:41:46,619 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44776
    2012-12-02 03:41:46,620 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.37
    2012-12-02 03:41:46,674 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0000 with negotiated timeout 40000 for
    client /10.40.222.77:44776
    2012-12-02 03:41:47,494 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5ac2ad7a0000, likely client has closed socket
    2012-12-02 03:41:47,495 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:44776 which had sessionid
    0x13b5ac2ad7a0000
    2012-12-02 03:42:16,001 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30003, likely client has closed socket
    2012-12-02 03:42:16,002 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.83.35.173:37153 which had sessionid
    0x13b5a963ea30003
    2012-12-02 03:42:17,916 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30002, likely client has closed socket
    2012-12-02 03:42:17,917 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:47487 which had sessionid
    0x13b5a963ea30002
    2012-12-02 03:42:19,468 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30001, likely client has closed socket
    2012-12-02 03:42:19,469 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:47488 which had sessionid
    0x13b5a963ea30001
    2012-12-02 03:42:28,000 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5ac2ad7a0000, timeout of 40000ms exceeded
    2012-12-02 03:42:28,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5ac2ad7a0000
    2012-12-02 03:42:46,214 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:52486
    2012-12-02 03:42:46,217 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:52486
    2012-12-02 03:42:46,220 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0001 with negotiated timeout 40000 for
    client /10.83.35.173:52486
    2012-12-02 03:42:47,857 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44799
    2012-12-02 03:42:47,864 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44799
    2012-12-02 03:42:47,867 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0002 with negotiated timeout 40000 for
    client /10.40.222.77:44799

    2012-12-02 03:42:47,991 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
    KeeperException when processing sessionid:0x13b5ac2ad7a0002 type:create
    cxid:0x6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
    Path:/hbase/master Error:KeeperErrorCode = NodeExists for /hbase/master
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30002, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30003, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30001, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30002
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30003
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30001
    2012-12-02 03:42:48,182 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44800
    2012-12-02 03:42:48,194 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44800
    2012-12-02 03:42:48,196 INFO org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0003 with negotiated timeout 40000 for
    client /10.40.222.77:44800
    Shouvanik,

    The errors look very much like
    http://zookeeper-user.578899.n2.nabble.com/Unusual-exception-td5632833.html

    Is your client on 10.40.222.77? It is connecting to ZK. Do you have any
    unusual network/firewall setting?

    Cheers,
    bc

    Regards,
    Shouvanik


    On Mon, Dec 3, 2012 at 1:44 AM, Harsh J wrote:

    Hi Shouvanik,

    Those logs are all INFO level and there's no issue in it. Can you
    share the issue/message you get when you mean "I am not able to start
    hbase" instead?

    On Mon, Dec 3, 2012 at 1:35 AM, Shouvanik Haldar
    wrote:
    Hi Harsh,

    I am not able to start hbase although hbase service is up. I am
    getting
    weird zookeeper exception. can you please help

    [root@ip-10-40-222-77 ~]# tail -f
    /var/log/zookeeper/zookeeper-cmf-zookeeper1-SERVER-ip-10-40-222-77.ec2.internal.log
    2012-12-02 03:42:47,991 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
    KeeperException when processing sessionid:0x13b5ac2ad7a0002
    type:create
    cxid:0x6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
    Path:/hbase/master Error:KeeperErrorCode = NodeExists for
    /hbase/master
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30002, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30003, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30001, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30002
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30003
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30001
    2012-12-02 03:42:48,182 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44800
    2012-12-02 03:42:48,194 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44800
    2012-12-02 03:42:48,196 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0003 with negotiated timeout 40000
    for
    client /10.40.222.77:44800


    Regards,
    Shouvanik Haldar


    On Mon, Dec 3, 2012 at 1:24 AM, Harsh J wrote:

    I replied to this over your duplicate post at

    https://groups.google.com/a/cloudera.org/d/topic/cdh-user/ft02DH52BZ4/discussion
    On Mon, Dec 3, 2012 at 12:59 AM, John H <john.hadoop@gmail.com>
    wrote:
    Hi,

    I installed Cloudera Manager(4.1.1) and CDH4 (4.1.2) using
    tarball
    on
    Ubuntu 10.04.4 (VM using virtual box). I was able to setup
    Cloudera
    manager
    server and agent, setup roles (NN,DN and Secondary NN) in pseudo
    distributed
    mode. I deployed the configuration to hosts and formatted name
    node
    successfully.

    When I started the services from Cloudera manager, I got message
    saying
    all
    3 (NN,DN,SNN) were started successfully, but I cannot access
    anyone
    of
    them
    through web url.
    The status for all 3 is started with good health.


    I looked at the log file for namenode and it is stuck at this
    line


    "2012-12-02 10:33:33,841 INFO
    org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager:
    Going
    to
    retain 2 images with txid >= 0"

    This is the first start after format, so I am confused why this
    is
    happening... Any help would be really appreciated..


    Thanks
    JH





    FULL LOG FILE: ***************************



    ************************************************************/
    2012-12-02 10:32:19,931 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting NameNode
    STARTUP_MSG: host = ubuntu1/127.0.0.1
    STARTUP_MSG: args = []
    STARTUP_MSG: version = 2.0.0-cdh4.1.2
    STARTUP_MSG: classpath =

    /home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-log4j12-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-json-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jettison-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jets3t-0.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-digester-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-annotations-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/activation-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/kfs-0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-net-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-auth-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsch-0.1.42.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-api-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/paranamer-2.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-math-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/junit-4.8.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/avro-1.7.1.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/stax-api-1.0.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-tests.jar:/home/john/cloudera-manager/cm-4.1.1/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-daemon-1.0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-tests.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-sources.jar:/usr/lib/hadoop-yarn/share/hadoop/yarn/*:/usr/lib/hadoop-mapreduce/share/hadoop/mapreduce/*
    STARTUP_MSG: build =

    file:///var/lib/jenkins/workspace/CDH4.1.2-Packaging-Hadoop/build/cdh4/hadoop/2.0.0-cdh4.1.2/source/hadoop-common-project/hadoop-common
    -r f0b53c81cbf56f5955e403b49fcd27afd5f082de; compiled by
    'jenkins' on
    Thu
    Nov 1 17:06:55 PDT 2012
    ************************************************************/
    2012-12-02 10:32:22,214 INFO
    org.apache.hadoop.metrics2.impl.MetricsConfig:
    loaded properties from hadoop-metrics2.properties
    2012-12-02 10:32:23,240 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled
    snapshot
    period
    at 10 second(s).
    2012-12-02 10:32:23,240 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode
    metrics
    system
    started
    2012-12-02 10:32:25,269 INFO
    org.apache.hadoop.util.HostsFileReader:
    Refreshing hosts (include/exclude) list
    2012-12-02 10:32:25,270 INFO
    org.apache.hadoop.util.HostsFileReader:
    Adding
    127.0.0.1 to the list of hosts from

    /home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE/dfs_hosts_allow.txt
    2012-12-02 10:32:25,362 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
    dfs.block.invalidate.limit=1000
    2012-12-02 10:32:25,422 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    dfs.block.access.token.enable=false
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    defaultReplication = 1
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    maxReplication
    = 512
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    minReplication
    = 1
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    maxReplicationStreams = 2
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    shouldCheckForEnoughRacks = true
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    replicationRecheckInterval = 3000
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    encryptDataTransfer = false
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
    =
    john (auth:SIMPLE)
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
    =
    john
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    isPermissionEnabled
    =
    true
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled:
    false
    2012-12-02 10:32:25,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append
    Enabled:
    true
    2012-12-02 10:32:27,741 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file
    names
    occuring
    more than 10 times
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.threshold-pct = 0.9990000128746033
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.min.datanodes = 0
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.extension = 30000
    2012-12-02 10:32:27,941 INFO
    org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /home/john/dfs_dir/namenode/data1/in_use.lock acquired by
    nodename
    2070@ubuntu1
    2012-12-02 10:32:28,574 INFO
    org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /home/john/dfs_dir/namenode/data2/in_use.lock acquired by
    nodename
    2070@ubuntu1
    2012-12-02 10:32:28,803 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Recovering
    unfinalized segments in /home/john/dfs_dir/namenode/data1/current
    2012-12-02 10:32:28,803 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Recovering
    unfinalized segments in /home/john/dfs_dir/namenode/data2/current
    2012-12-02 10:32:28,805 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    No edit log streams selected.
    2012-12-02 10:32:28,950 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Loading image file
    /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000
    using
    no compression
    2012-12-02 10:32:28,950 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Number of files = 1
    2012-12-02 10:32:28,951 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Number of files under construction = 0
    2012-12-02 10:32:28,952 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Image file of size 117 loaded in 0 seconds.
    2012-12-02 10:32:28,952 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Loaded image for txid 0 from
    /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000
    2012-12-02 10:32:28,955 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
    segment
    at 1
    2012-12-02 10:32:30,041 INFO
    org.apache.hadoop.hdfs.server.namenode.NameCache: initialized
    with 0
    entries
    0 lookups
    2012-12-02 10:32:30,041 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished
    loading
    FSImage in 2298 msecs
    2012-12-02 10:32:31,360 INFO org.apache.hadoop.ipc.Server:
    Starting
    Socket
    Reader #1 for port 8020
    2012-12-02 10:32:31,402 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
    FSNamesystemState MBean
    2012-12-02 10:32:31,482 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
    blocks
    under
    construction: 0
    2012-12-02 10:32:31,482 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: initializing
    replication queues
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Total
    number
    of
    blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number of
    invalid blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number of
    under-replicated blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number of
    over-replicated blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number of
    blocks
    being written = 0
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    Replication Queue initialization scan for invalid, over- and
    under-replicated blocks completed in 31 msec
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    Leaving safe mode after 6 secs.
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    Network topology has 0 racks and 0 datanodes
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    UnderReplicatedBlocks has 0 blocks
    2012-12-02 10:32:31,674 INFO org.mortbay.log: Logging to
    org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
    org.mortbay.log.Slf4jLog
    2012-12-02 10:32:31,814 INFO org.apache.hadoop.http.HttpServer:
    Added
    global
    filter 'safety'
    (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer:
    Added
    filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context hdfs
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer:
    Added
    filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context logs
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer:
    Added
    filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context static
    2012-12-02 10:32:31,841 INFO org.apache.hadoop.http.HttpServer:
    dfs.webhdfs.enabled = true
    2012-12-02 10:32:31,842 INFO org.apache.hadoop.http.HttpServer:
    Added
    filter
    'SPNEGO' (class=org.apache.hadoop.hdfs.web.AuthFilter)
    2012-12-02 10:32:31,844 INFO org.apache.hadoop.http.HttpServer:
    addJerseyResourcePackage:

    packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
    pathSpec=/webhdfs/v1/*
    2012-12-02 10:32:31,862 INFO org.apache.hadoop.http.HttpServer:
    Jetty
    bound
    to port 50070
    2012-12-02 10:32:31,862 INFO org.mortbay.log:
    jetty-6.1.26.cloudera.2
    2012-12-02 10:32:32,261 WARN
    org.apache.hadoop.security.authentication.server.AuthenticationFilter:
    'signature.secret' configuration not set, using a random value as
    secret
    2012-12-02 10:32:32,453 INFO org.mortbay.log: Started
    selectchannelconnector@localhost.localdomain:50070
    2012-12-02 10:32:32,453 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up
    at:
    localhost.localdomain:50070
    2012-12-02 10:32:32,453 INFO org.apache.hadoop.ipc.Server: IPC
    Server
    Responder: starting
    2012-12-02 10:32:32,454 INFO org.apache.hadoop.ipc.Server: IPC
    Server
    listener on 8020: starting
    2012-12-02 10:32:32,603 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode up at:
    ubuntu1/127.0.0.1:8020
    2012-12-02 10:32:32,603 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting
    services
    required for active state
    2012-12-02 10:32:34,359 INFO
    org.apache.hadoop.fs.TrashPolicyDefault:
    The
    configured checkpoint interval is 0 minutes. Using an interval of
    1440
    minutes that is used for deletion instead
    2012-12-02 10:32:35,780 INFO org.apache.hadoop.hdfs.StateChange:
    BLOCK*
    NameSystem.registerDatanode: node registration from
    DatanodeRegistration(127.0.0.1,
    storageID=DS-377461142-127.0.0.1-50010-1354473155123,
    infoPort=50075,
    ipcPort=50020,
    storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0)
    storage
    DS-377461142-127.0.0.1-50010-1354473155123
    2012-12-02 10:32:35,790 INFO
    org.apache.hadoop.net.NetworkTopology:
    Adding a
    new node: /default/127.0.0.1:50010
    2012-12-02 10:32:35,951 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    BLOCK*
    processReport: Received first block report from 127.0.0.1:50010after
    becoming active. Its block contents are no longer considered
    stale.
    2012-12-02 10:32:35,951 INFO org.apache.hadoop.hdfs.StateChange:
    BLOCK*
    processReport: from DatanodeRegistration(127.0.0.1,
    storageID=DS-377461142-127.0.0.1-50010-1354473155123,
    infoPort=50075,
    ipcPort=50020,
    storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0),
    blocks:
    0, processing time: 1 msecs
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit
    Log
    from
    127.0.0.1
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit
    logs.
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
    segment
    1
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
    transactions: 2
    Total time for transactions(ms): 1Number of transactions batched
    in
    Syncs: 0
    Number of syncs: 2 SyncTimes(ms): 166 189
    2012-12-02 10:33:31,595 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
    transactions: 2
    Total time for transactions(ms): 1Number of transactions batched
    in
    Syncs: 0
    Number of syncs: 3 SyncTimes(ms): 215 255
    2012-12-02 10:33:31,603 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Finalizing
    edits
    file

    /home/john/dfs_dir/namenode/data1/current/edits_inprogress_0000000000000000001
    ->

    /home/john/dfs_dir/namenode/data1/current/edits_0000000000000000001-0000000000000000002
    2012-12-02 10:33:31,604 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Finalizing
    edits
    file

    /home/john/dfs_dir/namenode/data2/current/edits_inprogress_0000000000000000001
    ->

    /home/john/dfs_dir/namenode/data2/current/edits_0000000000000000001-0000000000000000002
    2012-12-02 10:33:31,604 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
    segment
    at 3
    2012-12-02 10:33:33,274 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Opening
    connection
    to

    http://ubuntu1:50090/getimage?getimage=1&txid=2&storageInfo=-40:829593698:0:cluster2
    2012-12-02 10:33:33,622 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Transfer
    took
    0.34s
    at 0.00 KB/s
    2012-12-02 10:33:33,622 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage:
    Downloaded
    file
    fsimage.ckpt_0000000000000000002 size 117 bytes.
    2012-12-02 10:33:33,841 INFO
    org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager:
    Going
    to
    retain 2 images with txid >= 0



    --
    Harsh J



    --
    Thanks,
    Shouvanik


    --
    Harsh J



    --
    Thanks,
    Shouvanik


    --
    Harsh J

    --
    Thanks,
    *Shouvanik*
  • Shouvanik Haldar at Dec 3, 2012 at 9:40 am
    Hi,

    I performed the steps as you told me. Still I am facing the previously
    pasted ZK exception.
    Where can I find the zookeeper logs that cloudera manager writes to? In the
    admin console, I can find the log location as /var/log/zookeeper. Is this
    the only location?

    Please help
    On Mon, Dec 3, 2012 at 1:42 PM, bc Wong wrote:

    On Sun, Dec 2, 2012 at 11:18 PM, Shouvanik Haldar <
    shouvanik.haldar@gmail.com> wrote:
    Hi,

    Cloudera and all the services require some ports. These are enabled in
    AWS security group. Except that I am not aware of any firewall settings.

    Tell me one more thing. Does the /etc/hosts have anything to do?

    Let me paste the contents of this file.

    *127.0.0.1 localhost
    10.40.222.77 ip-10-40-222-77*


    and for the other server

    *127.0.0.1 localhost
    10.83.35.173 ip-10-83-35-173*

    Is this OK?
    Looks fine. Maybe you should heed this warning:

    "ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is
    able
    to connect to ZooKeeper but the connection closes immediately. This could
    be
    a sign that the server has too many connections (30 is the default).
    Consider inspecting your ZK server logs for that error and then make sure
    you are reusing HBaseConfiguration as often as you can. See HTable's
    javadoc
    for more information."

    Can you restart your ZK and HBase services, and them immediately try to do
    a "describe".

    Another help.
    Can you please point to the exact guide to upgrade to CM 4.x?
    https://ccp.cloudera.com/display/ENT4DOC/Upgrading+to+Cloudera+Manager+4.0

    See the subsections on upgrading from/to different versions.

    Cheers,
    bc

    Regards,
    Shouvanik



    On Mon, Dec 3, 2012 at 12:15 PM, bc Wong wrote:
    On Sun, Dec 2, 2012 at 12:34 PM, Harsh J wrote:

    Hi Shouvanik,

    I notice you use CDH3. You should really be using CDH4, as it provides
    a great deal of improvements over the CDH3 version that you'll be
    missing out on.

    In any case:

    If there exists a /etc/zookeeper/zoo.cfg on your client machine,
    please rename it to something else, such as zoo.cfg.unwanted, or
    remove it altogether, and retry.

    Alternatively, try to run your HBase shell command(s) with the direct
    script (instead of wrapper /usr/bin/hbase) as:

    /usr/lib/hbase/bin/hbase shell

    And see if it lets you connect and work normally this time.

    On Mon, Dec 3, 2012 at 1:50 AM, Shouvanik Haldar
    wrote:
    Yes Harsh,

    [root@ip-10-40-222-77 ~]# hbase shell
    HBase Shell; enter 'help<RETURN>' for list of supported commands.
    Type "exit<RETURN>" to leave the HBase Shell
    Version 0.90.4-cdh3u3, r, Thu Jan 26 10:13:36 PST 2012

    hbase(main):001:0> describe 't1'

    ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able
    to connect to ZooKeeper but the connection closes immediately. This could be
    a sign that the server has too many connections (30 is the default).
    Consider inspecting your ZK server logs for that error and then make sure
    you are reusing HBaseConfiguration as often as you can. See HTable's javadoc
    for more information.

    Here is some help for this command:
    Describe the named table. For example:
    hbase> describe 't1'


    hbase(main):002:0>


    Secondly,

    When I see the zookeeper logs, it says

    [root@ip-10-40-222-77 ~]# cat
    /var/log/zookeeper/zookeeper-cmf-zookeeper1-SERVER-ip-10-40-222-77.ec2.internal.log
    2012-12-01 23:30:35,726 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/1-zookeeper-server/zoo.cfg
    2012-12-01 23:30:35,731 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-01 23:30:35,811 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/1-zookeeper-server/zoo.cfg
    2012-12-01 23:30:35,812 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-01 23:30:35,842 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/1-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/1-zookeeper-server
    2012-12-01 23:30:35,851 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-01 23:30:35,851 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-01 23:30:35,851 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-01 23:30:35,891 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-01 23:30:35,907 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.0
    2012-12-01 23:30:35,941 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 28
    2012-12-01 23:38:21,260 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/9-zookeeper-server/zoo.cfg
    2012-12-01 23:38:21,265 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-01 23:38:21,339 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/9-zookeeper-server/zoo.cfg
    2012-12-01 23:38:21,340 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/9-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/9-zookeeper-server
    2012-12-01 23:38:21,366 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-01 23:38:21,366 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-01 23:38:21,366 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-01 23:38:21,399 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-01 23:38:21,415 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.28
    2012-12-01 23:38:21,430 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 28
    2012-12-01 23:43:49,121 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:45741
    2012-12-01 23:43:49,128 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:45741
    2012-12-01 23:43:49,130 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.29
    2012-12-01 23:43:49,172 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0000 with negotiated timeout 40000 for
    client /10.83.35.173:45741
    2012-12-01 23:43:52,894 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:59824
    2012-12-01 23:43:52,897 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:59824
    2012-12-01 23:43:52,899 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0001 with negotiated timeout 40000 for
    client /10.40.222.77:59824
    2012-12-01 23:43:52,993 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:59825
    2012-12-01 23:43:52,996 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:59825
    2012-12-01 23:43:52,998 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0002 with negotiated timeout 40000 for
    client /10.40.222.77:59825
    2012-12-02 02:48:13,910 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/34-zookeeper-server/zoo.cfg
    2012-12-02 02:48:13,914 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-02 02:48:14,002 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/34-zookeeper-server/zoo.cfg
    2012-12-02 02:48:14,003 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-02 02:48:14,030 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-02 02:48:14,030 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-02 02:48:14,030 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/34-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/34-zookeeper-server
    2012-12-02 02:48:14,039 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-02 02:48:14,039 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-02 02:48:14,040 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-02 02:48:14,072 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-02 02:48:14,088 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.28
    2012-12-02 02:48:14,109 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 2c
    2012-12-02 02:48:15,025 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:39458
    2012-12-02 02:48:15,029 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0002 at /
    10.40.222.77:39458
    2012-12-02 02:48:15,035 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0002 with negotiated timeout 40000 for
    client /10.40.222.77:39458
    2012-12-02 02:48:15,060 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:56157
    2012-12-02 02:48:15,062 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0000 at /
    10.83.35.173:56157
    2012-12-02 02:48:15,063 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0000 with negotiated timeout 40000 for
    client /10.83.35.173:56157
    2012-12-02 02:48:15,095 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:39459
    2012-12-02 02:48:15,095 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0001 at /
    10.40.222.77:39459
    2012-12-02 02:48:15,096 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0001 with negotiated timeout 40000 for
    client /10.40.222.77:39459
    2012-12-02 03:23:48,488 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.2d
    2012-12-02 03:23:48,560 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0001
    2012-12-02 03:23:48,570 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:39459 which had sessionid
    0x13b59e867fc0001
    2012-12-02 03:25:46,004 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54443
    2012-12-02 03:25:46,007 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54443
    2012-12-02 03:25:46,009 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30000 with negotiated timeout 40000 for
    client /10.40.222.77:54443
    2012-12-02 03:25:46,348 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30000, likely client has closed socket
    2012-12-02 03:25:46,349 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:54443 which had sessionid
    0x13b5a963ea30000
    2012-12-02 03:26:15,044 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b59e867fc0000, likely client has closed socket
    2012-12-02 03:26:15,045 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.83.35.173:56157 which had sessionid
    0x13b59e867fc0000
    2012-12-02 03:26:17,867 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b59e867fc0002, likely client has closed socket
    2012-12-02 03:26:17,867 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:39458 which had sessionid
    0x13b59e867fc0002
    2012-12-02 03:26:28,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30000, timeout of 40000ms exceeded
    2012-12-02 03:26:28,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30000
    2012-12-02 03:26:42,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b59e867fc0000, timeout of 40000ms exceeded
    2012-12-02 03:26:42,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0000
    2012-12-02 03:26:47,391 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54461
    2012-12-02 03:26:47,394 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54461
    2012-12-02 03:26:47,396 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30001 with negotiated timeout 40000 for
    client /10.40.222.77:54461
    2012-12-02 03:26:47,448 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54462
    2012-12-02 03:26:47,460 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54462
    2012-12-02 03:26:47,463 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30002 with negotiated timeout 40000 for
    client /10.40.222.77:54462
    2012-12-02 03:26:56,000 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b59e867fc0002, timeout of 40000ms exceeded
    2012-12-02 03:26:56,000 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0002
    2012-12-02 03:27:00,606 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:56794
    2012-12-02 03:27:00,614 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:56794
    2012-12-02 03:27:00,617 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30003 with negotiated timeout 40000 for
    client /10.83.35.173:56794
    2012-12-02 03:36:45,885 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/43-zookeeper-server/zoo.cfg
    2012-12-02 03:36:45,889 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-02 03:36:45,962 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/43-zookeeper-server/zoo.cfg
    2012-12-02 03:36:45,962 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/43-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/43-zookeeper-server
    2012-12-02 03:36:45,985 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-02 03:36:45,985 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-02 03:36:45,985 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-02 03:36:46,024 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-02 03:36:46,042 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.2c
    2012-12-02 03:36:46,063 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 36
    2012-12-02 03:36:46,531 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:37153
    2012-12-02 03:36:46,535 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30003 at /
    10.83.35.173:37153
    2012-12-02 03:36:46,539 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30003 with negotiated timeout 40000 for
    client /10.83.35.173:37153
    2012-12-02 03:36:46,883 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:47487
    2012-12-02 03:36:46,884 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30002 at /
    10.40.222.77:47487
    2012-12-02 03:36:46,884 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30002 with negotiated timeout 40000 for
    client /10.40.222.77:47487
    2012-12-02 03:36:47,310 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:47488
    2012-12-02 03:36:47,311 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30001 at /
    10.40.222.77:47488
    2012-12-02 03:36:47,312 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30001 with negotiated timeout 40000 for
    client /10.40.222.77:47488
    2012-12-02 03:41:46,616 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44776
    2012-12-02 03:41:46,619 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44776
    2012-12-02 03:41:46,620 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.37
    2012-12-02 03:41:46,674 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0000 with negotiated timeout 40000 for
    client /10.40.222.77:44776
    2012-12-02 03:41:47,494 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5ac2ad7a0000, likely client has closed socket
    2012-12-02 03:41:47,495 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:44776 which had sessionid
    0x13b5ac2ad7a0000
    2012-12-02 03:42:16,001 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30003, likely client has closed socket
    2012-12-02 03:42:16,002 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.83.35.173:37153 which had sessionid
    0x13b5a963ea30003
    2012-12-02 03:42:17,916 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30002, likely client has closed socket
    2012-12-02 03:42:17,917 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:47487 which had sessionid
    0x13b5a963ea30002
    2012-12-02 03:42:19,468 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30001, likely client has closed socket
    2012-12-02 03:42:19,469 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:47488 which had sessionid
    0x13b5a963ea30001
    2012-12-02 03:42:28,000 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5ac2ad7a0000, timeout of 40000ms exceeded
    2012-12-02 03:42:28,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5ac2ad7a0000
    2012-12-02 03:42:46,214 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:52486
    2012-12-02 03:42:46,217 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:52486
    2012-12-02 03:42:46,220 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0001 with negotiated timeout 40000 for
    client /10.83.35.173:52486
    2012-12-02 03:42:47,857 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44799
    2012-12-02 03:42:47,864 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44799
    2012-12-02 03:42:47,867 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0002 with negotiated timeout 40000 for
    client /10.40.222.77:44799

    2012-12-02 03:42:47,991 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
    KeeperException when processing sessionid:0x13b5ac2ad7a0002
    type:create
    cxid:0x6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
    Path:/hbase/master Error:KeeperErrorCode = NodeExists for
    /hbase/master
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30002, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30003, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30001, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30002
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30003
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30001
    2012-12-02 03:42:48,182 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44800
    2012-12-02 03:42:48,194 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44800
    2012-12-02 03:42:48,196 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0003 with negotiated timeout 40000 for
    client /10.40.222.77:44800
    Shouvanik,

    The errors look very much like
    http://zookeeper-user.578899.n2.nabble.com/Unusual-exception-td5632833.html

    Is your client on 10.40.222.77? It is connecting to ZK. Do you have any
    unusual network/firewall setting?

    Cheers,
    bc

    Regards,
    Shouvanik


    On Mon, Dec 3, 2012 at 1:44 AM, Harsh J wrote:

    Hi Shouvanik,

    Those logs are all INFO level and there's no issue in it. Can you
    share the issue/message you get when you mean "I am not able to start
    hbase" instead?

    On Mon, Dec 3, 2012 at 1:35 AM, Shouvanik Haldar
    wrote:
    Hi Harsh,

    I am not able to start hbase although hbase service is up. I am
    getting
    weird zookeeper exception. can you please help

    [root@ip-10-40-222-77 ~]# tail -f
    /var/log/zookeeper/zookeeper-cmf-zookeeper1-SERVER-ip-10-40-222-77.ec2.internal.log
    2012-12-02 03:42:47,991 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
    KeeperException when processing sessionid:0x13b5ac2ad7a0002
    type:create
    cxid:0x6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
    Path:/hbase/master Error:KeeperErrorCode = NodeExists for
    /hbase/master
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30002, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30003, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30001, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30002
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30003
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30001
    2012-12-02 03:42:48,182 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44800
    2012-12-02 03:42:48,194 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44800
    2012-12-02 03:42:48,196 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0003 with negotiated timeout
    40000 for
    client /10.40.222.77:44800


    Regards,
    Shouvanik Haldar



    On Mon, Dec 3, 2012 at 1:24 AM, Harsh J <harsh@cloudera.com>
    wrote:
    I replied to this over your duplicate post at

    https://groups.google.com/a/cloudera.org/d/topic/cdh-user/ft02DH52BZ4/discussion
    On Mon, Dec 3, 2012 at 12:59 AM, John H <john.hadoop@gmail.com>
    wrote:
    Hi,

    I installed Cloudera Manager(4.1.1) and CDH4 (4.1.2) using
    tarball
    on
    Ubuntu 10.04.4 (VM using virtual box). I was able to setup
    Cloudera
    manager
    server and agent, setup roles (NN,DN and Secondary NN) in pseudo
    distributed
    mode. I deployed the configuration to hosts and formatted name
    node
    successfully.

    When I started the services from Cloudera manager, I got message
    saying
    all
    3 (NN,DN,SNN) were started successfully, but I cannot access
    anyone
    of
    them
    through web url.
    The status for all 3 is started with good health.


    I looked at the log file for namenode and it is stuck at this
    line


    "2012-12-02 10:33:33,841 INFO
    org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager:
    Going
    to
    retain 2 images with txid >= 0"

    This is the first start after format, so I am confused why this
    is
    happening... Any help would be really appreciated..


    Thanks
    JH





    FULL LOG FILE: ***************************



    ************************************************************/
    2012-12-02 10:32:19,931 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting NameNode
    STARTUP_MSG: host = ubuntu1/127.0.0.1
    STARTUP_MSG: args = []
    STARTUP_MSG: version = 2.0.0-cdh4.1.2
    STARTUP_MSG: classpath =

    /home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-log4j12-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-json-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jettison-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jets3t-0.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-digester-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-annotations-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/activation-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/kfs-0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-net-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-auth-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsch-0.1.42.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-api-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/paranamer-2.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-math-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/junit-4.8.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/avro-1.7.1.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/stax-api-1.0.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-tests.jar:/home/john/cloudera-manager/cm-4.1.1/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-daemon-1.0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-tests.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-sources.jar:/usr/lib/hadoop-yarn/share/hadoop/yarn/*:/usr/lib/hadoop-mapreduce/share/hadoop/mapreduce/*
    STARTUP_MSG: build =

    file:///var/lib/jenkins/workspace/CDH4.1.2-Packaging-Hadoop/build/cdh4/hadoop/2.0.0-cdh4.1.2/source/hadoop-common-project/hadoop-common
    -r f0b53c81cbf56f5955e403b49fcd27afd5f082de; compiled by
    'jenkins' on
    Thu
    Nov 1 17:06:55 PDT 2012
    ************************************************************/
    2012-12-02 10:32:22,214 INFO
    org.apache.hadoop.metrics2.impl.MetricsConfig:
    loaded properties from hadoop-metrics2.properties
    2012-12-02 10:32:23,240 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled
    snapshot
    period
    at 10 second(s).
    2012-12-02 10:32:23,240 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode
    metrics
    system
    started
    2012-12-02 10:32:25,269 INFO
    org.apache.hadoop.util.HostsFileReader:
    Refreshing hosts (include/exclude) list
    2012-12-02 10:32:25,270 INFO
    org.apache.hadoop.util.HostsFileReader:
    Adding
    127.0.0.1 to the list of hosts from

    /home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE/dfs_hosts_allow.txt
    2012-12-02 10:32:25,362 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
    dfs.block.invalidate.limit=1000
    2012-12-02 10:32:25,422 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    dfs.block.access.token.enable=false
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    defaultReplication = 1
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    maxReplication
    = 512
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    minReplication
    = 1
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    maxReplicationStreams = 2
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    shouldCheckForEnoughRacks = true
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    replicationRecheckInterval = 3000
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    encryptDataTransfer = false
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
    =
    john (auth:SIMPLE)
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
    =
    john
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    isPermissionEnabled
    =
    true
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled:
    false
    2012-12-02 10:32:25,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append
    Enabled:
    true
    2012-12-02 10:32:27,741 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file
    names
    occuring
    more than 10 times
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.threshold-pct = 0.9990000128746033
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.min.datanodes = 0
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.extension = 30000
    2012-12-02 10:32:27,941 INFO
    org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /home/john/dfs_dir/namenode/data1/in_use.lock acquired
    by
    nodename
    2070@ubuntu1
    2012-12-02 10:32:28,574 INFO
    org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /home/john/dfs_dir/namenode/data2/in_use.lock acquired
    by
    nodename
    2070@ubuntu1
    2012-12-02 10:32:28,803 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Recovering
    unfinalized segments in
    /home/john/dfs_dir/namenode/data1/current
    2012-12-02 10:32:28,803 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Recovering
    unfinalized segments in
    /home/john/dfs_dir/namenode/data2/current
    2012-12-02 10:32:28,805 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    No edit log streams selected.
    2012-12-02 10:32:28,950 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Loading image file
    /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000
    using
    no compression
    2012-12-02 10:32:28,950 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Number of files = 1
    2012-12-02 10:32:28,951 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Number of files under construction = 0
    2012-12-02 10:32:28,952 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Image file of size 117 loaded in 0 seconds.
    2012-12-02 10:32:28,952 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Loaded image for txid 0 from
    /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000
    2012-12-02 10:32:28,955 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
    segment
    at 1
    2012-12-02 10:32:30,041 INFO
    org.apache.hadoop.hdfs.server.namenode.NameCache: initialized
    with 0
    entries
    0 lookups
    2012-12-02 10:32:30,041 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished
    loading
    FSImage in 2298 msecs
    2012-12-02 10:32:31,360 INFO org.apache.hadoop.ipc.Server:
    Starting
    Socket
    Reader #1 for port 8020
    2012-12-02 10:32:31,402 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
    FSNamesystemState MBean
    2012-12-02 10:32:31,482 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
    blocks
    under
    construction: 0
    2012-12-02 10:32:31,482 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    initializing
    replication queues
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Total
    number
    of
    blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number of
    invalid blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number of
    under-replicated blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number of
    over-replicated blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number of
    blocks
    being written = 0
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    Replication Queue initialization scan for invalid, over- and
    under-replicated blocks completed in 31 msec
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    Leaving safe mode after 6 secs.
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    Network topology has 0 racks and 0 datanodes
    2012-12-02 10:32:31,513 INFO org.apache.hadoop.hdfs.StateChange:
    STATE*
    UnderReplicatedBlocks has 0 blocks
    2012-12-02 10:32:31,674 INFO org.mortbay.log: Logging to
    org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
    org.mortbay.log.Slf4jLog
    2012-12-02 10:32:31,814 INFO org.apache.hadoop.http.HttpServer:
    Added
    global
    filter 'safety'
    (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer:
    Added
    filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context hdfs
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer:
    Added
    filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context logs
    2012-12-02 10:32:31,815 INFO org.apache.hadoop.http.HttpServer:
    Added
    filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context static
    2012-12-02 10:32:31,841 INFO org.apache.hadoop.http.HttpServer:
    dfs.webhdfs.enabled = true
    2012-12-02 10:32:31,842 INFO org.apache.hadoop.http.HttpServer:
    Added
    filter
    'SPNEGO' (class=org.apache.hadoop.hdfs.web.AuthFilter)
    2012-12-02 10:32:31,844 INFO org.apache.hadoop.http.HttpServer:
    addJerseyResourcePackage:

    packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
    pathSpec=/webhdfs/v1/*
    2012-12-02 10:32:31,862 INFO org.apache.hadoop.http.HttpServer:
    Jetty
    bound
    to port 50070
    2012-12-02 10:32:31,862 INFO org.mortbay.log:
    jetty-6.1.26.cloudera.2
    2012-12-02 10:32:32,261 WARN
    org.apache.hadoop.security.authentication.server.AuthenticationFilter:
    'signature.secret' configuration not set, using a random value
    as
    secret
    2012-12-02 10:32:32,453 INFO org.mortbay.log: Started
    selectchannelconnector@localhost.localdomain:50070
    2012-12-02 10:32:32,453 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up
    at:
    localhost.localdomain:50070
    2012-12-02 10:32:32,453 INFO org.apache.hadoop.ipc.Server: IPC
    Server
    Responder: starting
    2012-12-02 10:32:32,454 INFO org.apache.hadoop.ipc.Server: IPC
    Server
    listener on 8020: starting
    2012-12-02 10:32:32,603 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode up at:
    ubuntu1/127.0.0.1:8020
    2012-12-02 10:32:32,603 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting
    services
    required for active state
    2012-12-02 10:32:34,359 INFO
    org.apache.hadoop.fs.TrashPolicyDefault:
    The
    configured checkpoint interval is 0 minutes. Using an interval
    of
    1440
    minutes that is used for deletion instead
    2012-12-02 10:32:35,780 INFO org.apache.hadoop.hdfs.StateChange:
    BLOCK*
    NameSystem.registerDatanode: node registration from
    DatanodeRegistration(127.0.0.1,
    storageID=DS-377461142-127.0.0.1-50010-1354473155123,
    infoPort=50075,
    ipcPort=50020,
    storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0)
    storage
    DS-377461142-127.0.0.1-50010-1354473155123
    2012-12-02 10:32:35,790 INFO
    org.apache.hadoop.net.NetworkTopology:
    Adding a
    new node: /default/127.0.0.1:50010
    2012-12-02 10:32:35,951 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    BLOCK*
    processReport: Received first block report from 127.0.0.1:50010after
    becoming active. Its block contents are no longer considered
    stale.
    2012-12-02 10:32:35,951 INFO org.apache.hadoop.hdfs.StateChange:
    BLOCK*
    processReport: from DatanodeRegistration(127.0.0.1,
    storageID=DS-377461142-127.0.0.1-50010-1354473155123,
    infoPort=50075,
    ipcPort=50020,
    storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0),
    blocks:
    0, processing time: 1 msecs
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit
    Log
    from
    127.0.0.1
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit
    logs.
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
    segment
    1
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
    transactions: 2
    Total time for transactions(ms): 1Number of transactions
    batched in
    Syncs: 0
    Number of syncs: 2 SyncTimes(ms): 166 189
    2012-12-02 10:33:31,595 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
    transactions: 2
    Total time for transactions(ms): 1Number of transactions
    batched in
    Syncs: 0
    Number of syncs: 3 SyncTimes(ms): 215 255
    2012-12-02 10:33:31,603 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Finalizing
    edits
    file

    /home/john/dfs_dir/namenode/data1/current/edits_inprogress_0000000000000000001
    ->

    /home/john/dfs_dir/namenode/data1/current/edits_0000000000000000001-0000000000000000002
    2012-12-02 10:33:31,604 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Finalizing
    edits
    file

    /home/john/dfs_dir/namenode/data2/current/edits_inprogress_0000000000000000001
    ->

    /home/john/dfs_dir/namenode/data2/current/edits_0000000000000000001-0000000000000000002
    2012-12-02 10:33:31,604 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
    segment
    at 3
    2012-12-02 10:33:33,274 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Opening
    connection
    to

    http://ubuntu1:50090/getimage?getimage=1&txid=2&storageInfo=-40:829593698:0:cluster2
    2012-12-02 10:33:33,622 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage:
    Transfer took
    0.34s
    at 0.00 KB/s
    2012-12-02 10:33:33,622 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage:
    Downloaded
    file
    fsimage.ckpt_0000000000000000002 size 117 bytes.
    2012-12-02 10:33:33,841 INFO
    org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager:
    Going
    to
    retain 2 images with txid >= 0



    --
    Harsh J



    --
    Thanks,
    Shouvanik


    --
    Harsh J



    --
    Thanks,
    Shouvanik


    --
    Harsh J

    --
    Thanks,
    *Shouvanik*

    --
    Thanks,
    *Shouvanik*
  • bc Wong at Dec 3, 2012 at 7:29 pm

    On Mon, Dec 3, 2012 at 1:40 AM, Shouvanik Haldar wrote:

    Hi,

    I performed the steps as you told me. Still I am facing the previously
    pasted ZK exception.
    Where can I find the zookeeper logs that cloudera manager writes to? In
    the admin console, I can find the log location as /var/log/zookeeper. Is
    this the only location?

    It's in /var/log/zookeeper. Can you run `netstat -ntp | grep 2181' on your
    ZK node to see whether there are a lot of client connections?

    Cheers,
    bc

    On Mon, Dec 3, 2012 at 1:42 PM, bc Wong wrote:

    On Sun, Dec 2, 2012 at 11:18 PM, Shouvanik Haldar <
    shouvanik.haldar@gmail.com> wrote:
    Hi,

    Cloudera and all the services require some ports. These are enabled in
    AWS security group. Except that I am not aware of any firewall settings.

    Tell me one more thing. Does the /etc/hosts have anything to do?

    Let me paste the contents of this file.

    *127.0.0.1 localhost
    10.40.222.77 ip-10-40-222-77*


    and for the other server

    *127.0.0.1 localhost
    10.83.35.173 ip-10-83-35-173*

    Is this OK?
    Looks fine. Maybe you should heed this warning:

    "ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is
    able
    to connect to ZooKeeper but the connection closes immediately. This could
    be
    a sign that the server has too many connections (30 is the default).
    Consider inspecting your ZK server logs for that error and then make sure
    you are reusing HBaseConfiguration as often as you can. See HTable's
    javadoc
    for more information."

    Can you restart your ZK and HBase services, and them immediately try to
    do a "describe".

    Another help.
    Can you please point to the exact guide to upgrade to CM 4.x?
    https://ccp.cloudera.com/display/ENT4DOC/Upgrading+to+Cloudera+Manager+4.0

    See the subsections on upgrading from/to different versions.

    Cheers,
    bc

    Regards,
    Shouvanik



    On Mon, Dec 3, 2012 at 12:15 PM, bc Wong wrote:
    On Sun, Dec 2, 2012 at 12:34 PM, Harsh J wrote:

    Hi Shouvanik,

    I notice you use CDH3. You should really be using CDH4, as it provides
    a great deal of improvements over the CDH3 version that you'll be
    missing out on.

    In any case:

    If there exists a /etc/zookeeper/zoo.cfg on your client machine,
    please rename it to something else, such as zoo.cfg.unwanted, or
    remove it altogether, and retry.

    Alternatively, try to run your HBase shell command(s) with the direct
    script (instead of wrapper /usr/bin/hbase) as:

    /usr/lib/hbase/bin/hbase shell

    And see if it lets you connect and work normally this time.

    On Mon, Dec 3, 2012 at 1:50 AM, Shouvanik Haldar
    wrote:
    Yes Harsh,

    [root@ip-10-40-222-77 ~]# hbase shell
    HBase Shell; enter 'help<RETURN>' for list of supported commands.
    Type "exit<RETURN>" to leave the HBase Shell
    Version 0.90.4-cdh3u3, r, Thu Jan 26 10:13:36 PST 2012

    hbase(main):001:0> describe 't1'

    ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able
    to connect to ZooKeeper but the connection closes immediately. This could be
    a sign that the server has too many connections (30 is the default).
    Consider inspecting your ZK server logs for that error and then make sure
    you are reusing HBaseConfiguration as often as you can. See HTable's javadoc
    for more information.

    Here is some help for this command:
    Describe the named table. For example:
    hbase> describe 't1'


    hbase(main):002:0>


    Secondly,

    When I see the zookeeper logs, it says

    [root@ip-10-40-222-77 ~]# cat
    /var/log/zookeeper/zookeeper-cmf-zookeeper1-SERVER-ip-10-40-222-77.ec2.internal.log
    2012-12-01 23:30:35,726 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/1-zookeeper-server/zoo.cfg
    2012-12-01 23:30:35,731 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-01 23:30:35,811 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/1-zookeeper-server/zoo.cfg
    2012-12-01 23:30:35,812 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-01 23:30:35,842 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/1-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/1-zookeeper-server
    2012-12-01 23:30:35,851 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-01 23:30:35,851 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-01 23:30:35,851 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-01 23:30:35,891 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-01 23:30:35,907 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.0
    2012-12-01 23:30:35,941 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog:
    Snapshotting: 28
    2012-12-01 23:38:21,260 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/9-zookeeper-server/zoo.cfg
    2012-12-01 23:38:21,265 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-01 23:38:21,339 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/9-zookeeper-server/zoo.cfg
    2012-12-01 23:38:21,340 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/9-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/9-zookeeper-server
    2012-12-01 23:38:21,366 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-01 23:38:21,366 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-01 23:38:21,366 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-01 23:38:21,399 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-01 23:38:21,415 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.28
    2012-12-01 23:38:21,430 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog:
    Snapshotting: 28
    2012-12-01 23:43:49,121 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:45741
    2012-12-01 23:43:49,128 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:45741
    2012-12-01 23:43:49,130 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.29
    2012-12-01 23:43:49,172 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0000 with negotiated timeout 40000 for
    client /10.83.35.173:45741
    2012-12-01 23:43:52,894 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:59824
    2012-12-01 23:43:52,897 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:59824
    2012-12-01 23:43:52,899 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0001 with negotiated timeout 40000 for
    client /10.40.222.77:59824
    2012-12-01 23:43:52,993 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:59825
    2012-12-01 23:43:52,996 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:59825
    2012-12-01 23:43:52,998 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0002 with negotiated timeout 40000 for
    client /10.40.222.77:59825
    2012-12-02 02:48:13,910 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/34-zookeeper-server/zoo.cfg
    2012-12-02 02:48:13,914 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-02 02:48:14,002 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/34-zookeeper-server/zoo.cfg
    2012-12-02 02:48:14,003 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-02 02:48:14,030 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-02 02:48:14,030 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-02 02:48:14,030 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/34-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/34-zookeeper-server
    2012-12-02 02:48:14,039 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-02 02:48:14,039 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-02 02:48:14,040 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-02 02:48:14,072 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-02 02:48:14,088 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.28
    2012-12-02 02:48:14,109 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog:
    Snapshotting: 2c
    2012-12-02 02:48:15,025 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:39458
    2012-12-02 02:48:15,029 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0002 at /
    10.40.222.77:39458
    2012-12-02 02:48:15,035 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0002 with negotiated timeout 40000 for
    client /10.40.222.77:39458
    2012-12-02 02:48:15,060 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:56157
    2012-12-02 02:48:15,062 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0000 at /
    10.83.35.173:56157
    2012-12-02 02:48:15,063 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0000 with negotiated timeout 40000 for
    client /10.83.35.173:56157
    2012-12-02 02:48:15,095 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:39459
    2012-12-02 02:48:15,095 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0001 at /
    10.40.222.77:39459
    2012-12-02 02:48:15,096 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0001 with negotiated timeout 40000 for
    client /10.40.222.77:39459
    2012-12-02 03:23:48,488 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.2d
    2012-12-02 03:23:48,560 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0001
    2012-12-02 03:23:48,570 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:39459 which had sessionid
    0x13b59e867fc0001
    2012-12-02 03:25:46,004 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54443
    2012-12-02 03:25:46,007 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54443
    2012-12-02 03:25:46,009 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30000 with negotiated timeout 40000 for
    client /10.40.222.77:54443
    2012-12-02 03:25:46,348 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30000, likely client has closed socket
    2012-12-02 03:25:46,349 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:54443 which had sessionid
    0x13b5a963ea30000
    2012-12-02 03:26:15,044 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b59e867fc0000, likely client has closed socket
    2012-12-02 03:26:15,045 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.83.35.173:56157 which had sessionid
    0x13b59e867fc0000
    2012-12-02 03:26:17,867 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b59e867fc0002, likely client has closed socket
    2012-12-02 03:26:17,867 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:39458 which had sessionid
    0x13b59e867fc0002
    2012-12-02 03:26:28,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30000, timeout of 40000ms exceeded
    2012-12-02 03:26:28,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30000
    2012-12-02 03:26:42,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b59e867fc0000, timeout of 40000ms exceeded
    2012-12-02 03:26:42,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0000
    2012-12-02 03:26:47,391 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54461
    2012-12-02 03:26:47,394 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54461
    2012-12-02 03:26:47,396 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30001 with negotiated timeout 40000 for
    client /10.40.222.77:54461
    2012-12-02 03:26:47,448 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54462
    2012-12-02 03:26:47,460 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54462
    2012-12-02 03:26:47,463 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30002 with negotiated timeout 40000 for
    client /10.40.222.77:54462
    2012-12-02 03:26:56,000 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b59e867fc0002, timeout of 40000ms exceeded
    2012-12-02 03:26:56,000 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0002
    2012-12-02 03:27:00,606 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:56794
    2012-12-02 03:27:00,614 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:56794
    2012-12-02 03:27:00,617 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30003 with negotiated timeout 40000 for
    client /10.83.35.173:56794
    2012-12-02 03:36:45,885 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/43-zookeeper-server/zoo.cfg
    2012-12-02 03:36:45,889 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-02 03:36:45,962 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/43-zookeeper-server/zoo.cfg
    2012-12-02 03:36:45,962 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/43-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/43-zookeeper-server
    2012-12-02 03:36:45,985 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-02 03:36:45,985 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-02 03:36:45,985 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-02 03:36:46,024 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-02 03:36:46,042 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.2c
    2012-12-02 03:36:46,063 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog:
    Snapshotting: 36
    2012-12-02 03:36:46,531 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:37153
    2012-12-02 03:36:46,535 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30003 at /
    10.83.35.173:37153
    2012-12-02 03:36:46,539 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30003 with negotiated timeout 40000 for
    client /10.83.35.173:37153
    2012-12-02 03:36:46,883 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:47487
    2012-12-02 03:36:46,884 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30002 at /
    10.40.222.77:47487
    2012-12-02 03:36:46,884 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30002 with negotiated timeout 40000 for
    client /10.40.222.77:47487
    2012-12-02 03:36:47,310 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:47488
    2012-12-02 03:36:47,311 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30001 at /
    10.40.222.77:47488
    2012-12-02 03:36:47,312 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30001 with negotiated timeout 40000 for
    client /10.40.222.77:47488
    2012-12-02 03:41:46,616 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44776
    2012-12-02 03:41:46,619 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44776
    2012-12-02 03:41:46,620 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.37
    2012-12-02 03:41:46,674 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0000 with negotiated timeout 40000 for
    client /10.40.222.77:44776
    2012-12-02 03:41:47,494 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5ac2ad7a0000, likely client has closed socket
    2012-12-02 03:41:47,495 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:44776 which had sessionid
    0x13b5ac2ad7a0000
    2012-12-02 03:42:16,001 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30003, likely client has closed socket
    2012-12-02 03:42:16,002 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.83.35.173:37153 which had sessionid
    0x13b5a963ea30003
    2012-12-02 03:42:17,916 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30002, likely client has closed socket
    2012-12-02 03:42:17,917 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:47487 which had sessionid
    0x13b5a963ea30002
    2012-12-02 03:42:19,468 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30001, likely client has closed socket
    2012-12-02 03:42:19,469 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:47488 which had sessionid
    0x13b5a963ea30001
    2012-12-02 03:42:28,000 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5ac2ad7a0000, timeout of 40000ms exceeded
    2012-12-02 03:42:28,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5ac2ad7a0000
    2012-12-02 03:42:46,214 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:52486
    2012-12-02 03:42:46,217 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:52486
    2012-12-02 03:42:46,220 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0001 with negotiated timeout 40000 for
    client /10.83.35.173:52486
    2012-12-02 03:42:47,857 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44799
    2012-12-02 03:42:47,864 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44799
    2012-12-02 03:42:47,867 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0002 with negotiated timeout 40000 for
    client /10.40.222.77:44799

    2012-12-02 03:42:47,991 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
    KeeperException when processing sessionid:0x13b5ac2ad7a0002
    type:create
    cxid:0x6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
    Path:/hbase/master Error:KeeperErrorCode = NodeExists for
    /hbase/master
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30002, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30003, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30001, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30002
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30003
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30001
    2012-12-02 03:42:48,182 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44800
    2012-12-02 03:42:48,194 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44800
    2012-12-02 03:42:48,196 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0003 with negotiated timeout 40000 for
    client /10.40.222.77:44800
    Shouvanik,

    The errors look very much like
    http://zookeeper-user.578899.n2.nabble.com/Unusual-exception-td5632833.html

    Is your client on 10.40.222.77? It is connecting to ZK. Do you have any
    unusual network/firewall setting?

    Cheers,
    bc

    Regards,
    Shouvanik


    On Mon, Dec 3, 2012 at 1:44 AM, Harsh J wrote:

    Hi Shouvanik,

    Those logs are all INFO level and there's no issue in it. Can you
    share the issue/message you get when you mean "I am not able to
    start
    hbase" instead?

    On Mon, Dec 3, 2012 at 1:35 AM, Shouvanik Haldar
    wrote:
    Hi Harsh,

    I am not able to start hbase although hbase service is up. I am
    getting
    weird zookeeper exception. can you please help

    [root@ip-10-40-222-77 ~]# tail -f
    /var/log/zookeeper/zookeeper-cmf-zookeeper1-SERVER-ip-10-40-222-77.ec2.internal.log
    2012-12-02 03:42:47,991 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
    KeeperException when processing sessionid:0x13b5ac2ad7a0002
    type:create
    cxid:0x6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
    Path:/hbase/master Error:KeeperErrorCode = NodeExists for
    /hbase/master
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30002, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30003, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30001, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed
    session
    termination for sessionid: 0x13b5a963ea30002
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed
    session
    termination for sessionid: 0x13b5a963ea30003
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed
    session
    termination for sessionid: 0x13b5a963ea30001
    2012-12-02 03:42:48,182 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44800
    2012-12-02 03:42:48,194 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44800
    2012-12-02 03:42:48,196 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0003 with negotiated timeout
    40000 for
    client /10.40.222.77:44800


    Regards,
    Shouvanik Haldar



    On Mon, Dec 3, 2012 at 1:24 AM, Harsh J <harsh@cloudera.com>
    wrote:
    I replied to this over your duplicate post at

    https://groups.google.com/a/cloudera.org/d/topic/cdh-user/ft02DH52BZ4/discussion
    On Mon, Dec 3, 2012 at 12:59 AM, John H <john.hadoop@gmail.com>
    wrote:
    Hi,

    I installed Cloudera Manager(4.1.1) and CDH4 (4.1.2) using
    tarball
    on
    Ubuntu 10.04.4 (VM using virtual box). I was able to setup
    Cloudera
    manager
    server and agent, setup roles (NN,DN and Secondary NN) in
    pseudo
    distributed
    mode. I deployed the configuration to hosts and formatted name
    node
    successfully.

    When I started the services from Cloudera manager, I got
    message
    saying
    all
    3 (NN,DN,SNN) were started successfully, but I cannot access
    anyone
    of
    them
    through web url.
    The status for all 3 is started with good health.


    I looked at the log file for namenode and it is stuck at this
    line


    "2012-12-02 10:33:33,841 INFO
    org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager:
    Going
    to
    retain 2 images with txid >= 0"

    This is the first start after format, so I am confused why
    this is
    happening... Any help would be really appreciated..


    Thanks
    JH





    FULL LOG FILE: ***************************



    ************************************************************/
    2012-12-02 10:32:19,931 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting NameNode
    STARTUP_MSG: host = ubuntu1/127.0.0.1
    STARTUP_MSG: args = []
    STARTUP_MSG: version = 2.0.0-cdh4.1.2
    STARTUP_MSG: classpath =

    /home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-log4j12-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-json-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jettison-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jets3t-0.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-digester-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-annotations-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/activation-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/kfs-0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-net-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-auth-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsch-0.1.42.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-api-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/paranamer-2.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-math-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/junit-4.8.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/avro-1.7.1.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/stax-api-1.0.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-tests.jar:/home/john/cloudera-manager/cm-4.1.1/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-daemon-1.0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-tests.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-sources.jar:/usr/lib/hadoop-yarn/share/hadoop/yarn/*:/usr/lib/hadoop-mapreduce/share/hadoop/mapreduce/*
    STARTUP_MSG: build =

    file:///var/lib/jenkins/workspace/CDH4.1.2-Packaging-Hadoop/build/cdh4/hadoop/2.0.0-cdh4.1.2/source/hadoop-common-project/hadoop-common
    -r f0b53c81cbf56f5955e403b49fcd27afd5f082de; compiled by
    'jenkins' on
    Thu
    Nov 1 17:06:55 PDT 2012
    ************************************************************/
    2012-12-02 10:32:22,214 INFO
    org.apache.hadoop.metrics2.impl.MetricsConfig:
    loaded properties from hadoop-metrics2.properties
    2012-12-02 10:32:23,240 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled
    snapshot
    period
    at 10 second(s).
    2012-12-02 10:32:23,240 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode
    metrics
    system
    started
    2012-12-02 10:32:25,269 INFO
    org.apache.hadoop.util.HostsFileReader:
    Refreshing hosts (include/exclude) list
    2012-12-02 10:32:25,270 INFO
    org.apache.hadoop.util.HostsFileReader:
    Adding
    127.0.0.1 to the list of hosts from

    /home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE/dfs_hosts_allow.txt
    2012-12-02 10:32:25,362 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
    dfs.block.invalidate.limit=1000
    2012-12-02 10:32:25,422 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    dfs.block.access.token.enable=false
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    defaultReplication = 1
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    maxReplication
    = 512
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    minReplication
    = 1
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    maxReplicationStreams = 2
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    shouldCheckForEnoughRacks = true
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    replicationRecheckInterval = 3000
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    encryptDataTransfer = false
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
    =
    john (auth:SIMPLE)
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
    =
    john
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    isPermissionEnabled
    =
    true
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA
    Enabled:
    false
    2012-12-02 10:32:25,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append
    Enabled:
    true
    2012-12-02 10:32:27,741 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file
    names
    occuring
    more than 10 times
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.threshold-pct = 0.9990000128746033
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.min.datanodes = 0
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.extension = 30000
    2012-12-02 10:32:27,941 INFO
    org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /home/john/dfs_dir/namenode/data1/in_use.lock acquired
    by
    nodename
    2070@ubuntu1
    2012-12-02 10:32:28,574 INFO
    org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /home/john/dfs_dir/namenode/data2/in_use.lock acquired
    by
    nodename
    2070@ubuntu1
    2012-12-02 10:32:28,803 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Recovering
    unfinalized segments in
    /home/john/dfs_dir/namenode/data1/current
    2012-12-02 10:32:28,803 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Recovering
    unfinalized segments in
    /home/john/dfs_dir/namenode/data2/current
    2012-12-02 10:32:28,805 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    No edit log streams selected.
    2012-12-02 10:32:28,950 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Loading image file
    /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000
    using
    no compression
    2012-12-02 10:32:28,950 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Number of files = 1
    2012-12-02 10:32:28,951 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Number of files under construction = 0
    2012-12-02 10:32:28,952 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Image file of size 117 loaded in 0 seconds.
    2012-12-02 10:32:28,952 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Loaded image for txid 0 from
    /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000
    2012-12-02 10:32:28,955 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
    segment
    at 1
    2012-12-02 10:32:30,041 INFO
    org.apache.hadoop.hdfs.server.namenode.NameCache: initialized
    with 0
    entries
    0 lookups
    2012-12-02 10:32:30,041 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished
    loading
    FSImage in 2298 msecs
    2012-12-02 10:32:31,360 INFO org.apache.hadoop.ipc.Server:
    Starting
    Socket
    Reader #1 for port 8020
    2012-12-02 10:32:31,402 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
    FSNamesystemState MBean
    2012-12-02 10:32:31,482 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
    blocks
    under
    construction: 0
    2012-12-02 10:32:31,482 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    initializing
    replication queues
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Total
    number
    of
    blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number of
    invalid blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number of
    under-replicated blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number of
    over-replicated blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number of
    blocks
    being written = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.StateChange:
    STATE*
    Replication Queue initialization scan for invalid, over- and
    under-replicated blocks completed in 31 msec
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.StateChange:
    STATE*
    Leaving safe mode after 6 secs.
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.StateChange:
    STATE*
    Network topology has 0 racks and 0 datanodes
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.StateChange:
    STATE*
    UnderReplicatedBlocks has 0 blocks
    2012-12-02 10:32:31,674 INFO org.mortbay.log: Logging to
    org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
    org.mortbay.log.Slf4jLog
    2012-12-02 10:32:31,814 INFO
    org.apache.hadoop.http.HttpServer: Added
    global
    filter 'safety'
    (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    2012-12-02 10:32:31,815 INFO
    org.apache.hadoop.http.HttpServer: Added
    filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context hdfs
    2012-12-02 10:32:31,815 INFO
    org.apache.hadoop.http.HttpServer: Added
    filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context logs
    2012-12-02 10:32:31,815 INFO
    org.apache.hadoop.http.HttpServer: Added
    filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context static
    2012-12-02 10:32:31,841 INFO org.apache.hadoop.http.HttpServer:
    dfs.webhdfs.enabled = true
    2012-12-02 10:32:31,842 INFO
    org.apache.hadoop.http.HttpServer: Added
    filter
    'SPNEGO' (class=org.apache.hadoop.hdfs.web.AuthFilter)
    2012-12-02 10:32:31,844 INFO org.apache.hadoop.http.HttpServer:
    addJerseyResourcePackage:

    packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
    pathSpec=/webhdfs/v1/*
    2012-12-02 10:32:31,862 INFO
    org.apache.hadoop.http.HttpServer: Jetty
    bound
    to port 50070
    2012-12-02 10:32:31,862 INFO org.mortbay.log:
    jetty-6.1.26.cloudera.2
    2012-12-02 10:32:32,261 WARN
    org.apache.hadoop.security.authentication.server.AuthenticationFilter:
    'signature.secret' configuration not set, using a random value
    as
    secret
    2012-12-02 10:32:32,453 INFO org.mortbay.log: Started
    selectchannelconnector@localhost.localdomain:50070
    2012-12-02 10:32:32,453 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up
    at:
    localhost.localdomain:50070
    2012-12-02 10:32:32,453 INFO org.apache.hadoop.ipc.Server: IPC
    Server
    Responder: starting
    2012-12-02 10:32:32,454 INFO org.apache.hadoop.ipc.Server: IPC
    Server
    listener on 8020: starting
    2012-12-02 10:32:32,603 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode up
    at:
    ubuntu1/127.0.0.1:8020
    2012-12-02 10:32:32,603 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting
    services
    required for active state
    2012-12-02 10:32:34,359 INFO
    org.apache.hadoop.fs.TrashPolicyDefault:
    The
    configured checkpoint interval is 0 minutes. Using an interval
    of
    1440
    minutes that is used for deletion instead
    2012-12-02 10:32:35,780 INFO
    org.apache.hadoop.hdfs.StateChange:
    BLOCK*
    NameSystem.registerDatanode: node registration from
    DatanodeRegistration(127.0.0.1,
    storageID=DS-377461142-127.0.0.1-50010-1354473155123,
    infoPort=50075,
    ipcPort=50020,
    storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0)
    storage
    DS-377461142-127.0.0.1-50010-1354473155123
    2012-12-02 10:32:35,790 INFO
    org.apache.hadoop.net.NetworkTopology:
    Adding a
    new node: /default/127.0.0.1:50010
    2012-12-02 10:32:35,951 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    BLOCK*
    processReport: Received first block report from
    127.0.0.1:50010 after
    becoming active. Its block contents are no longer considered
    stale.
    2012-12-02 10:32:35,951 INFO
    org.apache.hadoop.hdfs.StateChange:
    BLOCK*
    processReport: from DatanodeRegistration(127.0.0.1,
    storageID=DS-377461142-127.0.0.1-50010-1354473155123,
    infoPort=50075,
    ipcPort=50020,
    storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0),
    blocks:
    0, processing time: 1 msecs
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit
    Log
    from
    127.0.0.1
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit
    logs.
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
    segment
    1
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
    transactions: 2
    Total time for transactions(ms): 1Number of transactions
    batched in
    Syncs: 0
    Number of syncs: 2 SyncTimes(ms): 166 189
    2012-12-02 10:33:31,595 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
    transactions: 2
    Total time for transactions(ms): 1Number of transactions
    batched in
    Syncs: 0
    Number of syncs: 3 SyncTimes(ms): 215 255
    2012-12-02 10:33:31,603 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Finalizing
    edits
    file

    /home/john/dfs_dir/namenode/data1/current/edits_inprogress_0000000000000000001
    ->

    /home/john/dfs_dir/namenode/data1/current/edits_0000000000000000001-0000000000000000002
    2012-12-02 10:33:31,604 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Finalizing
    edits
    file

    /home/john/dfs_dir/namenode/data2/current/edits_inprogress_0000000000000000001
    ->

    /home/john/dfs_dir/namenode/data2/current/edits_0000000000000000001-0000000000000000002
    2012-12-02 10:33:31,604 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
    segment
    at 3
    2012-12-02 10:33:33,274 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Opening
    connection
    to

    http://ubuntu1:50090/getimage?getimage=1&txid=2&storageInfo=-40:829593698:0:cluster2
    2012-12-02 10:33:33,622 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage:
    Transfer took
    0.34s
    at 0.00 KB/s
    2012-12-02 10:33:33,622 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage:
    Downloaded
    file
    fsimage.ckpt_0000000000000000002 size 117 bytes.
    2012-12-02 10:33:33,841 INFO
    org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager:
    Going
    to
    retain 2 images with txid >= 0



    --
    Harsh J



    --
    Thanks,
    Shouvanik


    --
    Harsh J



    --
    Thanks,
    Shouvanik


    --
    Harsh J

    --
    Thanks,
    *Shouvanik*

    --
    Thanks,
    *Shouvanik*
  • Shouvanik Haldar at Dec 4, 2012 at 5:02 pm
    Hi,

    I did that and below is the list of client connections

    tcp 0 0 ::ffff:10.40.222.77:2181 ::ffff:10.40.222.77:43512
    ESTABLISHED 20441/java
    tcp 0 0 ::ffff:10.40.222.77:2181 ::ffff:10.40.222.77:43513
    ESTABLISHED 20441/java
    tcp 0 0 ::ffff:10.40.222.77:43512 ::ffff:10.40.222.77:2181
    ESTABLISHED 22681/java
    tcp 0 0 ::ffff:10.40.222.77:43513 ::ffff:10.40.222.77:2181
    ESTABLISHED 22707/java
    tcp 0 0 ::ffff:10.40.222.77:51510 ::ffff:10.40.222.77:2181
    TIME_WAIT -

    Is that a problem?

    Please help me folks...I am not able to execute simple hbase query..since I
    am not able to connect to HBase...below I paste all logs that I think may
    give some hint why I am not able to achieve success




    2012-12-04 06:34:56,057 ERROR
    org.apache.hadoop.hbase.master.HMasterCommandLine: Master not running

    ERROR org.apache.hadoop.hdfs.DFSClient: Exception closing file
    /hbase/hbase.version : org.apache.hadoop.ipc.RemoteException:
    java.io.IOException: File /hbase/hbase.version could only be replicated to
    0 nodes, instead of 1

    2012-12-04 09:56:33,009 INFO
    org.apache.hadoop.hbase.master.ActiveMasterManager: Master=
    ec2-75-101-165-36.compute-1.amazonaws.com:60000
    2012-12-04 09:56:33,438 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
    Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
    /hbase/hbase.version could only be replicated to 0 nodes, instead of 1


    2012-12-04 09:56:43,450 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
    Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
    /hbase/hbase.version could only be replicated to 0 nodes, instead of 1

    2012-12-04 09:57:43,533 WARN org.apache.hadoop.hdfs.DFSClient: Error
    Recovery for block null bad datanode[0] nodes == null





    12/12/04 11:13:06 ERROR zookeeper.ZKConfig: no valid quorum servers found
    in zoo.cfg

    ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: An error is
    preventing HBase from connecting to ZooKeeper

    2012-12-04 11:19:06,584 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b66b4d47c0000, likely client has closed socket

    2012-12-04 11:20:14,000 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b65bbe40c0013, timeout of 40000ms exceeded

    Got user-level KeeperException when processing sessionid:0x13b66b4d47c0001
    type:create cxid:0x6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
    Error Path:/hbase/master Error:KeeperErrorCode = NodeExists for
    /hbase/master



    ERROR: org.apache.hadoop.hbase.MasterNotRunningException:
    ec2-75-101-165-36.compute-1.amazonaws.com:60000



    On Tue, Dec 4, 2012 at 12:58 AM, bc Wong wrote:

    On Mon, Dec 3, 2012 at 1:40 AM, Shouvanik Haldar <
    shouvanik.haldar@gmail.com> wrote:
    Hi,

    I performed the steps as you told me. Still I am facing the previously
    pasted ZK exception.
    Where can I find the zookeeper logs that cloudera manager writes to? In
    the admin console, I can find the log location as /var/log/zookeeper. Is
    this the only location?

    It's in /var/log/zookeeper. Can you run `netstat -ntp | grep 2181' on your
    ZK node to see whether there are a lot of client connections?

    Cheers,
    bc

    On Mon, Dec 3, 2012 at 1:42 PM, bc Wong wrote:

    On Sun, Dec 2, 2012 at 11:18 PM, Shouvanik Haldar <
    shouvanik.haldar@gmail.com> wrote:
    Hi,

    Cloudera and all the services require some ports. These are enabled in
    AWS security group. Except that I am not aware of any firewall settings.

    Tell me one more thing. Does the /etc/hosts have anything to do?

    Let me paste the contents of this file.

    *127.0.0.1 localhost
    10.40.222.77 ip-10-40-222-77*


    and for the other server

    *127.0.0.1 localhost
    10.83.35.173 ip-10-83-35-173*

    Is this OK?
    Looks fine. Maybe you should heed this warning:

    "ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is
    able
    to connect to ZooKeeper but the connection closes immediately. This
    could be
    a sign that the server has too many connections (30 is the default).
    Consider inspecting your ZK server logs for that error and then make sure
    you are reusing HBaseConfiguration as often as you can. See HTable's
    javadoc
    for more information."

    Can you restart your ZK and HBase services, and them immediately try to
    do a "describe".

    Another help.
    Can you please point to the exact guide to upgrade to CM 4.x?

    https://ccp.cloudera.com/display/ENT4DOC/Upgrading+to+Cloudera+Manager+4.0

    See the subsections on upgrading from/to different versions.

    Cheers,
    bc

    Regards,
    Shouvanik



    On Mon, Dec 3, 2012 at 12:15 PM, bc Wong wrote:
    On Sun, Dec 2, 2012 at 12:34 PM, Harsh J wrote:

    Hi Shouvanik,

    I notice you use CDH3. You should really be using CDH4, as it provides
    a great deal of improvements over the CDH3 version that you'll be
    missing out on.

    In any case:

    If there exists a /etc/zookeeper/zoo.cfg on your client machine,
    please rename it to something else, such as zoo.cfg.unwanted, or
    remove it altogether, and retry.

    Alternatively, try to run your HBase shell command(s) with the direct
    script (instead of wrapper /usr/bin/hbase) as:

    /usr/lib/hbase/bin/hbase shell

    And see if it lets you connect and work normally this time.

    On Mon, Dec 3, 2012 at 1:50 AM, Shouvanik Haldar
    wrote:
    Yes Harsh,

    [root@ip-10-40-222-77 ~]# hbase shell
    HBase Shell; enter 'help<RETURN>' for list of supported commands.
    Type "exit<RETURN>" to leave the HBase Shell
    Version 0.90.4-cdh3u3, r, Thu Jan 26 10:13:36 PST 2012

    hbase(main):001:0> describe 't1'

    ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able
    to connect to ZooKeeper but the connection closes immediately. This could be
    a sign that the server has too many connections (30 is the default).
    Consider inspecting your ZK server logs for that error and then make sure
    you are reusing HBaseConfiguration as often as you can. See
    HTable's javadoc
    for more information.

    Here is some help for this command:
    Describe the named table. For example:
    hbase> describe 't1'


    hbase(main):002:0>


    Secondly,

    When I see the zookeeper logs, it says

    [root@ip-10-40-222-77 ~]# cat
    /var/log/zookeeper/zookeeper-cmf-zookeeper1-SERVER-ip-10-40-222-77.ec2.internal.log
    2012-12-01 23:30:35,726 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/1-zookeeper-server/zoo.cfg
    2012-12-01 23:30:35,731 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-01 23:30:35,811 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/1-zookeeper-server/zoo.cfg
    2012-12-01 23:30:35,812 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-01 23:30:35,842 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/1-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/1-zookeeper-server
    2012-12-01 23:30:35,851 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-01 23:30:35,851 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-01 23:30:35,851 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-01 23:30:35,891 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-01 23:30:35,907 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.0
    2012-12-01 23:30:35,941 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog:
    Snapshotting: 28
    2012-12-01 23:38:21,260 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/9-zookeeper-server/zoo.cfg
    2012-12-01 23:38:21,265 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-01 23:38:21,339 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from: /var/run/cloudera-scm-agent/process/9-zookeeper-server/zoo.cfg
    2012-12-01 23:38:21,340 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/9-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/9-zookeeper-server
    2012-12-01 23:38:21,366 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-01 23:38:21,366 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-01 23:38:21,366 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-01 23:38:21,399 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-01 23:38:21,415 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.28
    2012-12-01 23:38:21,430 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog:
    Snapshotting: 28
    2012-12-01 23:43:49,121 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:45741
    2012-12-01 23:43:49,128 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:45741
    2012-12-01 23:43:49,130 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.29
    2012-12-01 23:43:49,172 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0000 with negotiated timeout 40000 for
    client /10.83.35.173:45741
    2012-12-01 23:43:52,894 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:59824
    2012-12-01 23:43:52,897 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:59824
    2012-12-01 23:43:52,899 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0001 with negotiated timeout 40000 for
    client /10.40.222.77:59824
    2012-12-01 23:43:52,993 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:59825
    2012-12-01 23:43:52,996 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:59825
    2012-12-01 23:43:52,998 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0002 with negotiated timeout 40000 for
    client /10.40.222.77:59825
    2012-12-02 02:48:13,910 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from:
    /var/run/cloudera-scm-agent/process/34-zookeeper-server/zoo.cfg
    2012-12-02 02:48:13,914 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-02 02:48:14,002 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from:
    /var/run/cloudera-scm-agent/process/34-zookeeper-server/zoo.cfg
    2012-12-02 02:48:14,003 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-02 02:48:14,030 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-02 02:48:14,030 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-02 02:48:14,030 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/34-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/34-zookeeper-server
    2012-12-02 02:48:14,039 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-02 02:48:14,039 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-02 02:48:14,040 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-02 02:48:14,072 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-02 02:48:14,088 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.28
    2012-12-02 02:48:14,109 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog:
    Snapshotting: 2c
    2012-12-02 02:48:15,025 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:39458
    2012-12-02 02:48:15,029 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0002 at /
    10.40.222.77:39458
    2012-12-02 02:48:15,035 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0002 with negotiated timeout 40000 for
    client /10.40.222.77:39458
    2012-12-02 02:48:15,060 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:56157
    2012-12-02 02:48:15,062 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0000 at /
    10.83.35.173:56157
    2012-12-02 02:48:15,063 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0000 with negotiated timeout 40000 for
    client /10.83.35.173:56157
    2012-12-02 02:48:15,095 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:39459
    2012-12-02 02:48:15,095 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0001 at /
    10.40.222.77:39459
    2012-12-02 02:48:15,096 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0001 with negotiated timeout 40000 for
    client /10.40.222.77:39459
    2012-12-02 03:23:48,488 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.2d
    2012-12-02 03:23:48,560 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0001
    2012-12-02 03:23:48,570 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:39459 which had sessionid
    0x13b59e867fc0001
    2012-12-02 03:25:46,004 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54443
    2012-12-02 03:25:46,007 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54443
    2012-12-02 03:25:46,009 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30000 with negotiated timeout 40000 for
    client /10.40.222.77:54443
    2012-12-02 03:25:46,348 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30000, likely client has closed socket
    2012-12-02 03:25:46,349 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:54443 which had sessionid
    0x13b5a963ea30000
    2012-12-02 03:26:15,044 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b59e867fc0000, likely client has closed socket
    2012-12-02 03:26:15,045 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.83.35.173:56157 which had sessionid
    0x13b59e867fc0000
    2012-12-02 03:26:17,867 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b59e867fc0002, likely client has closed socket
    2012-12-02 03:26:17,867 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:39458 which had sessionid
    0x13b59e867fc0002
    2012-12-02 03:26:28,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30000, timeout of 40000ms exceeded
    2012-12-02 03:26:28,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30000
    2012-12-02 03:26:42,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b59e867fc0000, timeout of 40000ms exceeded
    2012-12-02 03:26:42,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0000
    2012-12-02 03:26:47,391 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54461
    2012-12-02 03:26:47,394 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54461
    2012-12-02 03:26:47,396 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30001 with negotiated timeout 40000 for
    client /10.40.222.77:54461
    2012-12-02 03:26:47,448 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54462
    2012-12-02 03:26:47,460 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54462
    2012-12-02 03:26:47,463 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30002 with negotiated timeout 40000 for
    client /10.40.222.77:54462
    2012-12-02 03:26:56,000 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b59e867fc0002, timeout of 40000ms exceeded
    2012-12-02 03:26:56,000 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0002
    2012-12-02 03:27:00,606 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:56794
    2012-12-02 03:27:00,614 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:56794
    2012-12-02 03:27:00,617 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30003 with negotiated timeout 40000 for
    client /10.83.35.173:56794
    2012-12-02 03:36:45,885 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from:
    /var/run/cloudera-scm-agent/process/43-zookeeper-server/zoo.cfg
    2012-12-02 03:36:45,889 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no config or no
    quorum defined in config, running in standalone mode
    2012-12-02 03:36:45,962 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from:
    /var/run/cloudera-scm-agent/process/43-zookeeper-server/zoo.cfg
    2012-12-02 03:36:45,962 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/43-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/43-zookeeper-server
    2012-12-02 03:36:45,985 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-02 03:36:45,985 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-02 03:36:45,985 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-02 03:36:46,024 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-02 03:36:46,042 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.2c
    2012-12-02 03:36:46,063 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog:
    Snapshotting: 36
    2012-12-02 03:36:46,531 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:37153
    2012-12-02 03:36:46,535 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30003 at /
    10.83.35.173:37153
    2012-12-02 03:36:46,539 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30003 with negotiated timeout 40000 for
    client /10.83.35.173:37153
    2012-12-02 03:36:46,883 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:47487
    2012-12-02 03:36:46,884 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30002 at /
    10.40.222.77:47487
    2012-12-02 03:36:46,884 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30002 with negotiated timeout 40000 for
    client /10.40.222.77:47487
    2012-12-02 03:36:47,310 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:47488
    2012-12-02 03:36:47,311 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30001 at /
    10.40.222.77:47488
    2012-12-02 03:36:47,312 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30001 with negotiated timeout 40000 for
    client /10.40.222.77:47488
    2012-12-02 03:41:46,616 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44776
    2012-12-02 03:41:46,619 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44776
    2012-12-02 03:41:46,620 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.37
    2012-12-02 03:41:46,674 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0000 with negotiated timeout 40000 for
    client /10.40.222.77:44776
    2012-12-02 03:41:47,494 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5ac2ad7a0000, likely client has closed socket
    2012-12-02 03:41:47,495 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:44776 which had sessionid
    0x13b5ac2ad7a0000
    2012-12-02 03:42:16,001 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30003, likely client has closed socket
    2012-12-02 03:42:16,002 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.83.35.173:37153 which had sessionid
    0x13b5a963ea30003
    2012-12-02 03:42:17,916 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30002, likely client has closed socket
    2012-12-02 03:42:17,917 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:47487 which had sessionid
    0x13b5a963ea30002
    2012-12-02 03:42:19,468 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30001, likely client has closed socket
    2012-12-02 03:42:19,469 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:47488 which had sessionid
    0x13b5a963ea30001
    2012-12-02 03:42:28,000 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5ac2ad7a0000, timeout of 40000ms exceeded
    2012-12-02 03:42:28,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5ac2ad7a0000
    2012-12-02 03:42:46,214 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:52486
    2012-12-02 03:42:46,217 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:52486
    2012-12-02 03:42:46,220 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0001 with negotiated timeout 40000 for
    client /10.83.35.173:52486
    2012-12-02 03:42:47,857 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44799
    2012-12-02 03:42:47,864 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44799
    2012-12-02 03:42:47,867 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0002 with negotiated timeout 40000 for
    client /10.40.222.77:44799

    2012-12-02 03:42:47,991 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
    KeeperException when processing sessionid:0x13b5ac2ad7a0002
    type:create
    cxid:0x6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
    Path:/hbase/master Error:KeeperErrorCode = NodeExists for
    /hbase/master
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30002, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30003, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30001, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30002
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30003
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30001
    2012-12-02 03:42:48,182 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44800
    2012-12-02 03:42:48,194 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44800
    2012-12-02 03:42:48,196 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0003 with negotiated timeout 40000 for
    client /10.40.222.77:44800
    Shouvanik,

    The errors look very much like
    http://zookeeper-user.578899.n2.nabble.com/Unusual-exception-td5632833.html

    Is your client on 10.40.222.77? It is connecting to ZK. Do you have
    any unusual network/firewall setting?

    Cheers,
    bc

    Regards,
    Shouvanik


    On Mon, Dec 3, 2012 at 1:44 AM, Harsh J wrote:

    Hi Shouvanik,

    Those logs are all INFO level and there's no issue in it. Can you
    share the issue/message you get when you mean "I am not able to
    start
    hbase" instead?

    On Mon, Dec 3, 2012 at 1:35 AM, Shouvanik Haldar
    wrote:
    Hi Harsh,

    I am not able to start hbase although hbase service is up. I am
    getting
    weird zookeeper exception. can you please help

    [root@ip-10-40-222-77 ~]# tail -f
    /var/log/zookeeper/zookeeper-cmf-zookeeper1-SERVER-ip-10-40-222-77.ec2.internal.log
    2012-12-02 03:42:47,991 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
    KeeperException when processing sessionid:0x13b5ac2ad7a0002
    type:create
    cxid:0x6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
    Error
    Path:/hbase/master Error:KeeperErrorCode = NodeExists for
    /hbase/master
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30002, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30003, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30001, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed
    session
    termination for sessionid: 0x13b5a963ea30002
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed
    session
    termination for sessionid: 0x13b5a963ea30003
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed
    session
    termination for sessionid: 0x13b5a963ea30001
    2012-12-02 03:42:48,182 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44800
    2012-12-02 03:42:48,194 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /
    10.40.222.77:44800
    2012-12-02 03:42:48,196 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0003 with negotiated timeout
    40000 for
    client /10.40.222.77:44800


    Regards,
    Shouvanik Haldar



    On Mon, Dec 3, 2012 at 1:24 AM, Harsh J <harsh@cloudera.com>
    wrote:
    I replied to this over your duplicate post at

    https://groups.google.com/a/cloudera.org/d/topic/cdh-user/ft02DH52BZ4/discussion
    On Mon, Dec 3, 2012 at 12:59 AM, John H <john.hadoop@gmail.com>
    wrote:
    Hi,

    I installed Cloudera Manager(4.1.1) and CDH4 (4.1.2) using
    tarball
    on
    Ubuntu 10.04.4 (VM using virtual box). I was able to setup
    Cloudera
    manager
    server and agent, setup roles (NN,DN and Secondary NN) in
    pseudo
    distributed
    mode. I deployed the configuration to hosts and formatted
    name node
    successfully.

    When I started the services from Cloudera manager, I got
    message
    saying
    all
    3 (NN,DN,SNN) were started successfully, but I cannot access
    anyone
    of
    them
    through web url.
    The status for all 3 is started with good health.


    I looked at the log file for namenode and it is stuck at this
    line


    "2012-12-02 10:33:33,841 INFO
    org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager:
    Going
    to
    retain 2 images with txid >= 0"

    This is the first start after format, so I am confused why
    this is
    happening... Any help would be really appreciated..


    Thanks
    JH





    FULL LOG FILE: ***************************



    ************************************************************/
    2012-12-02 10:32:19,931 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting NameNode
    STARTUP_MSG: host = ubuntu1/127.0.0.1
    STARTUP_MSG: args = []
    STARTUP_MSG: version = 2.0.0-cdh4.1.2
    STARTUP_MSG: classpath =

    /home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-log4j12-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-json-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jettison-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jets3t-0.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-digester-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-annotations-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/activation-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/kfs-0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-net-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-auth-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsch-0.1.42.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-api-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/paranamer-2.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-math-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/junit-4.8.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/avro-1.7.1.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/stax-api-1.0.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-tests.jar:/home/john/cloudera-manager/cm-4.1.1/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-daemon-1.0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-tests.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-sources.jar:/usr/lib/hadoop-yarn/share/hadoop/yarn/*:/usr/lib/hadoop-mapreduce/share/hadoop/mapreduce/*
    STARTUP_MSG: build =

    file:///var/lib/jenkins/workspace/CDH4.1.2-Packaging-Hadoop/build/cdh4/hadoop/2.0.0-cdh4.1.2/source/hadoop-common-project/hadoop-common
    -r f0b53c81cbf56f5955e403b49fcd27afd5f082de; compiled by
    'jenkins' on
    Thu
    Nov 1 17:06:55 PDT 2012
    ************************************************************/
    2012-12-02 10:32:22,214 INFO
    org.apache.hadoop.metrics2.impl.MetricsConfig:
    loaded properties from hadoop-metrics2.properties
    2012-12-02 10:32:23,240 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled
    snapshot
    period
    at 10 second(s).
    2012-12-02 10:32:23,240 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode
    metrics
    system
    started
    2012-12-02 10:32:25,269 INFO
    org.apache.hadoop.util.HostsFileReader:
    Refreshing hosts (include/exclude) list
    2012-12-02 10:32:25,270 INFO
    org.apache.hadoop.util.HostsFileReader:
    Adding
    127.0.0.1 to the list of hosts from

    /home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE/dfs_hosts_allow.txt
    2012-12-02 10:32:25,362 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
    dfs.block.invalidate.limit=1000
    2012-12-02 10:32:25,422 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    dfs.block.access.token.enable=false
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    defaultReplication = 1
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    maxReplication
    = 512
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    minReplication
    = 1
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    maxReplicationStreams = 2
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    shouldCheckForEnoughRacks = true
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    replicationRecheckInterval = 3000
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    encryptDataTransfer = false
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
    =
    john (auth:SIMPLE)
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    supergroup
    =
    john
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    isPermissionEnabled
    =
    true
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA
    Enabled:
    false
    2012-12-02 10:32:25,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append
    Enabled:
    true
    2012-12-02 10:32:27,741 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file
    names
    occuring
    more than 10 times
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.threshold-pct = 0.9990000128746033
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.min.datanodes = 0
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.extension = 30000
    2012-12-02 10:32:27,941 INFO
    org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /home/john/dfs_dir/namenode/data1/in_use.lock
    acquired by
    nodename
    2070@ubuntu1
    2012-12-02 10:32:28,574 INFO
    org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /home/john/dfs_dir/namenode/data2/in_use.lock
    acquired by
    nodename
    2070@ubuntu1
    2012-12-02 10:32:28,803 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Recovering
    unfinalized segments in
    /home/john/dfs_dir/namenode/data1/current
    2012-12-02 10:32:28,803 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Recovering
    unfinalized segments in
    /home/john/dfs_dir/namenode/data2/current
    2012-12-02 10:32:28,805 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    No edit log streams selected.
    2012-12-02 10:32:28,950 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Loading image file
    /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000
    using
    no compression
    2012-12-02 10:32:28,950 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Number of files = 1
    2012-12-02 10:32:28,951 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Number of files under construction = 0
    2012-12-02 10:32:28,952 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Image file of size 117 loaded in 0 seconds.
    2012-12-02 10:32:28,952 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Loaded image for txid 0 from
    /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000
    2012-12-02 10:32:28,955 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
    segment
    at 1
    2012-12-02 10:32:30,041 INFO
    org.apache.hadoop.hdfs.server.namenode.NameCache: initialized
    with 0
    entries
    0 lookups
    2012-12-02 10:32:30,041 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished
    loading
    FSImage in 2298 msecs
    2012-12-02 10:32:31,360 INFO org.apache.hadoop.ipc.Server:
    Starting
    Socket
    Reader #1 for port 8020
    2012-12-02 10:32:31,402 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    Registered
    FSNamesystemState MBean
    2012-12-02 10:32:31,482 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number
    of blocks
    under
    construction: 0
    2012-12-02 10:32:31,482 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    initializing
    replication queues
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Total
    number
    of
    blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number of
    invalid blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number of
    under-replicated blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number of
    over-replicated blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number of
    blocks
    being written = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.StateChange:
    STATE*
    Replication Queue initialization scan for invalid, over- and
    under-replicated blocks completed in 31 msec
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.StateChange:
    STATE*
    Leaving safe mode after 6 secs.
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.StateChange:
    STATE*
    Network topology has 0 racks and 0 datanodes
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.StateChange:
    STATE*
    UnderReplicatedBlocks has 0 blocks
    2012-12-02 10:32:31,674 INFO org.mortbay.log: Logging to
    org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
    org.mortbay.log.Slf4jLog
    2012-12-02 10:32:31,814 INFO
    org.apache.hadoop.http.HttpServer: Added
    global
    filter 'safety'
    (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    2012-12-02 10:32:31,815 INFO
    org.apache.hadoop.http.HttpServer: Added
    filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context hdfs
    2012-12-02 10:32:31,815 INFO
    org.apache.hadoop.http.HttpServer: Added
    filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context logs
    2012-12-02 10:32:31,815 INFO
    org.apache.hadoop.http.HttpServer: Added
    filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context static
    2012-12-02 10:32:31,841 INFO
    org.apache.hadoop.http.HttpServer:
    dfs.webhdfs.enabled = true
    2012-12-02 10:32:31,842 INFO
    org.apache.hadoop.http.HttpServer: Added
    filter
    'SPNEGO' (class=org.apache.hadoop.hdfs.web.AuthFilter)
    2012-12-02 10:32:31,844 INFO
    org.apache.hadoop.http.HttpServer:
    addJerseyResourcePackage:

    packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
    pathSpec=/webhdfs/v1/*
    2012-12-02 10:32:31,862 INFO
    org.apache.hadoop.http.HttpServer: Jetty
    bound
    to port 50070
    2012-12-02 10:32:31,862 INFO org.mortbay.log:
    jetty-6.1.26.cloudera.2
    2012-12-02 10:32:32,261 WARN
    org.apache.hadoop.security.authentication.server.AuthenticationFilter:
    'signature.secret' configuration not set, using a random
    value as
    secret
    2012-12-02 10:32:32,453 INFO org.mortbay.log: Started
    selectchannelconnector@localhost.localdomain:50070
    2012-12-02 10:32:32,453 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server
    up at:
    localhost.localdomain:50070
    2012-12-02 10:32:32,453 INFO org.apache.hadoop.ipc.Server:
    IPC Server
    Responder: starting
    2012-12-02 10:32:32,454 INFO org.apache.hadoop.ipc.Server:
    IPC Server
    listener on 8020: starting
    2012-12-02 10:32:32,603 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode up
    at:
    ubuntu1/127.0.0.1:8020
    2012-12-02 10:32:32,603 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting
    services
    required for active state
    2012-12-02 10:32:34,359 INFO
    org.apache.hadoop.fs.TrashPolicyDefault:
    The
    configured checkpoint interval is 0 minutes. Using an
    interval of
    1440
    minutes that is used for deletion instead
    2012-12-02 10:32:35,780 INFO
    org.apache.hadoop.hdfs.StateChange:
    BLOCK*
    NameSystem.registerDatanode: node registration from
    DatanodeRegistration(127.0.0.1,
    storageID=DS-377461142-127.0.0.1-50010-1354473155123,
    infoPort=50075,
    ipcPort=50020,
    storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0)
    storage
    DS-377461142-127.0.0.1-50010-1354473155123
    2012-12-02 10:32:35,790 INFO
    org.apache.hadoop.net.NetworkTopology:
    Adding a
    new node: /default/127.0.0.1:50010
    2012-12-02 10:32:35,951 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    BLOCK*
    processReport: Received first block report from
    127.0.0.1:50010 after
    becoming active. Its block contents are no longer considered
    stale.
    2012-12-02 10:32:35,951 INFO
    org.apache.hadoop.hdfs.StateChange:
    BLOCK*
    processReport: from DatanodeRegistration(127.0.0.1,
    storageID=DS-377461142-127.0.0.1-50010-1354473155123,
    infoPort=50075,
    ipcPort=50020,
    storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0),
    blocks:
    0, processing time: 1 msecs
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll
    Edit Log
    from
    127.0.0.1
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling
    edit logs.
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
    segment
    1
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
    transactions: 2
    Total time for transactions(ms): 1Number of transactions
    batched in
    Syncs: 0
    Number of syncs: 2 SyncTimes(ms): 166 189
    2012-12-02 10:33:31,595 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
    transactions: 2
    Total time for transactions(ms): 1Number of transactions
    batched in
    Syncs: 0
    Number of syncs: 3 SyncTimes(ms): 215 255
    2012-12-02 10:33:31,603 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Finalizing
    edits
    file

    /home/john/dfs_dir/namenode/data1/current/edits_inprogress_0000000000000000001
    ->

    /home/john/dfs_dir/namenode/data1/current/edits_0000000000000000001-0000000000000000002
    2012-12-02 10:33:31,604 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Finalizing
    edits
    file

    /home/john/dfs_dir/namenode/data2/current/edits_inprogress_0000000000000000001
    ->

    /home/john/dfs_dir/namenode/data2/current/edits_0000000000000000001-0000000000000000002
    2012-12-02 10:33:31,604 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
    segment
    at 3
    2012-12-02 10:33:33,274 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage:
    Opening
    connection
    to

    http://ubuntu1:50090/getimage?getimage=1&txid=2&storageInfo=-40:829593698:0:cluster2
    2012-12-02 10:33:33,622 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage:
    Transfer took
    0.34s
    at 0.00 KB/s
    2012-12-02 10:33:33,622 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage:
    Downloaded
    file
    fsimage.ckpt_0000000000000000002 size 117 bytes.
    2012-12-02 10:33:33,841 INFO
    org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager:
    Going
    to
    retain 2 images with txid >= 0



    --
    Harsh J



    --
    Thanks,
    Shouvanik


    --
    Harsh J



    --
    Thanks,
    Shouvanik


    --
    Harsh J

    --
    Thanks,
    *Shouvanik*

    --
    Thanks,
    *Shouvanik*

    --
    Thanks,
    *Shouvanik*
  • bc Wong at Dec 4, 2012 at 5:56 pm

    On Tue, Dec 4, 2012 at 9:02 AM, Shouvanik Haldar wrote:
    Hi,

    I did that and below is the list of client connections

    tcp 0 0 ::ffff:10.40.222.77:2181 ::ffff:10.40.222.77:43512
    ESTABLISHED 20441/java
    tcp 0 0 ::ffff:10.40.222.77:2181 ::ffff:10.40.222.77:43513
    ESTABLISHED 20441/java
    tcp 0 0 ::ffff:10.40.222.77:43512 ::ffff:10.40.222.77:2181
    ESTABLISHED 22681/java
    tcp 0 0 ::ffff:10.40.222.77:43513 ::ffff:10.40.222.77:2181
    ESTABLISHED 22707/java
    tcp 0 0 ::ffff:10.40.222.77:51510 ::ffff:10.40.222.77:2181
    TIME_WAIT -

    Is that a problem?
    Looks ok.

    Please help me folks...I am not able to execute simple hbase query..since
    I am not able to connect to HBase...below I paste all logs that I think may
    give some hint why I am not able to achieve success




    2012-12-04 06:34:56,057 ERROR
    org.apache.hadoop.hbase.master.HMasterCommandLine: Master not running

    ERROR org.apache.hadoop.hdfs.DFSClient: Exception closing file
    /hbase/hbase.version : org.apache.hadoop.ipc.RemoteException:
    java.io.IOException: File /hbase/hbase.version could only be replicated to
    0 nodes, instead of 1
    This error is new (for your cluster). Can you go to your HDFS daemons and
    look for errors in the logs? Can you successfully do a simple `hadoop fs
    -touchz /tmp/foo'?

    Cheers,
    bc

    2012-12-04 09:56:33,009 INFO
    org.apache.hadoop.hbase.master.ActiveMasterManager: Master=
    ec2-75-101-165-36.compute-1.amazonaws.com:60000
    2012-12-04 09:56:33,438 WARN org.apache.hadoop.hdfs.DFSClient:
    DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
    java.io.IOException: File /hbase/hbase.version could only be replicated to
    0 nodes, instead of 1


    2012-12-04 09:56:43,450 WARN org.apache.hadoop.hdfs.DFSClient:
    DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
    java.io.IOException: File /hbase/hbase.version could only be replicated to
    0 nodes, instead of 1

    2012-12-04 09:57:43,533 WARN org.apache.hadoop.hdfs.DFSClient: Error
    Recovery for block null bad datanode[0] nodes == null





    12/12/04 11:13:06 ERROR zookeeper.ZKConfig: no valid quorum servers found
    in zoo.cfg

    ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: An error is
    preventing HBase from connecting to ZooKeeper

    2012-12-04 11:19:06,584 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b66b4d47c0000, likely client has closed socket

    2012-12-04 11:20:14,000 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b65bbe40c0013, timeout of 40000ms exceeded

    Got user-level KeeperException when processing sessionid:0x13b66b4d47c0001
    type:create cxid:0x6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
    Error Path:/hbase/master Error:KeeperErrorCode = NodeExists for
    /hbase/master



    ERROR: org.apache.hadoop.hbase.MasterNotRunningException:
    ec2-75-101-165-36.compute-1.amazonaws.com:60000



    On Tue, Dec 4, 2012 at 12:58 AM, bc Wong wrote:

    On Mon, Dec 3, 2012 at 1:40 AM, Shouvanik Haldar <
    shouvanik.haldar@gmail.com> wrote:
    Hi,

    I performed the steps as you told me. Still I am facing the previously
    pasted ZK exception.
    Where can I find the zookeeper logs that cloudera manager writes to? In
    the admin console, I can find the log location as /var/log/zookeeper. Is
    this the only location?

    It's in /var/log/zookeeper. Can you run `netstat -ntp | grep 2181' on
    your ZK node to see whether there are a lot of client connections?

    Cheers,
    bc

    On Mon, Dec 3, 2012 at 1:42 PM, bc Wong wrote:

    On Sun, Dec 2, 2012 at 11:18 PM, Shouvanik Haldar <
    shouvanik.haldar@gmail.com> wrote:
    Hi,

    Cloudera and all the services require some ports. These are enabled in
    AWS security group. Except that I am not aware of any firewall settings.

    Tell me one more thing. Does the /etc/hosts have anything to do?

    Let me paste the contents of this file.

    *127.0.0.1 localhost
    10.40.222.77 ip-10-40-222-77*


    and for the other server

    *127.0.0.1 localhost
    10.83.35.173 ip-10-83-35-173*

    Is this OK?
    Looks fine. Maybe you should heed this warning:

    "ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is
    able
    to connect to ZooKeeper but the connection closes immediately. This
    could be
    a sign that the server has too many connections (30 is the default).
    Consider inspecting your ZK server logs for that error and then make
    sure
    you are reusing HBaseConfiguration as often as you can. See HTable's
    javadoc
    for more information."

    Can you restart your ZK and HBase services, and them immediately try to
    do a "describe".

    Another help.
    Can you please point to the exact guide to upgrade to CM 4.x?

    https://ccp.cloudera.com/display/ENT4DOC/Upgrading+to+Cloudera+Manager+4.0

    See the subsections on upgrading from/to different versions.

    Cheers,
    bc

    Regards,
    Shouvanik



    On Mon, Dec 3, 2012 at 12:15 PM, bc Wong wrote:
    On Sun, Dec 2, 2012 at 12:34 PM, Harsh J wrote:

    Hi Shouvanik,

    I notice you use CDH3. You should really be using CDH4, as it
    provides
    a great deal of improvements over the CDH3 version that you'll be
    missing out on.

    In any case:

    If there exists a /etc/zookeeper/zoo.cfg on your client machine,
    please rename it to something else, such as zoo.cfg.unwanted, or
    remove it altogether, and retry.

    Alternatively, try to run your HBase shell command(s) with the direct
    script (instead of wrapper /usr/bin/hbase) as:

    /usr/lib/hbase/bin/hbase shell

    And see if it lets you connect and work normally this time.

    On Mon, Dec 3, 2012 at 1:50 AM, Shouvanik Haldar
    wrote:
    Yes Harsh,

    [root@ip-10-40-222-77 ~]# hbase shell
    HBase Shell; enter 'help<RETURN>' for list of supported commands.
    Type "exit<RETURN>" to leave the HBase Shell
    Version 0.90.4-cdh3u3, r, Thu Jan 26 10:13:36 PST 2012

    hbase(main):001:0> describe 't1'

    ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able
    to connect to ZooKeeper but the connection closes immediately.
    This could be
    a sign that the server has too many connections (30 is the default).
    Consider inspecting your ZK server logs for that error and then make sure
    you are reusing HBaseConfiguration as often as you can. See
    HTable's javadoc
    for more information.

    Here is some help for this command:
    Describe the named table. For example:
    hbase> describe 't1'


    hbase(main):002:0>


    Secondly,

    When I see the zookeeper logs, it says

    [root@ip-10-40-222-77 ~]# cat
    /var/log/zookeeper/zookeeper-cmf-zookeeper1-SERVER-ip-10-40-222-77.ec2.internal.log
    2012-12-01 23:30:35,726 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from:
    /var/run/cloudera-scm-agent/process/1-zookeeper-server/zoo.cfg
    2012-12-01 23:30:35,731 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no
    config or no
    quorum defined in config, running in standalone mode
    2012-12-01 23:30:35,811 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from:
    /var/run/cloudera-scm-agent/process/1-zookeeper-server/zoo.cfg
    2012-12-01 23:30:35,812 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-01 23:30:35,842 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/1-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/1-zookeeper-server
    2012-12-01 23:30:35,851 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-01 23:30:35,851 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-01 23:30:35,851 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-01 23:30:35,891 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-01 23:30:35,907 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.0
    2012-12-01 23:30:35,941 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog:
    Snapshotting: 28
    2012-12-01 23:38:21,260 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from:
    /var/run/cloudera-scm-agent/process/9-zookeeper-server/zoo.cfg
    2012-12-01 23:38:21,265 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no
    config or no
    quorum defined in config, running in standalone mode
    2012-12-01 23:38:21,339 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from:
    /var/run/cloudera-scm-agent/process/9-zookeeper-server/zoo.cfg
    2012-12-01 23:38:21,340 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/9-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/9-zookeeper-server
    2012-12-01 23:38:21,366 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-01 23:38:21,366 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-01 23:38:21,366 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-01 23:38:21,399 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-01 23:38:21,415 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.28
    2012-12-01 23:38:21,430 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog:
    Snapshotting: 28
    2012-12-01 23:43:49,121 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:45741
    2012-12-01 23:43:49,128 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:45741
    2012-12-01 23:43:49,130 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.29
    2012-12-01 23:43:49,172 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0000 with negotiated timeout 40000 for
    client /10.83.35.173:45741
    2012-12-01 23:43:52,894 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:59824
    2012-12-01 23:43:52,897 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:59824
    2012-12-01 23:43:52,899 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0001 with negotiated timeout 40000 for
    client /10.40.222.77:59824
    2012-12-01 23:43:52,993 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:59825
    2012-12-01 23:43:52,996 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:59825
    2012-12-01 23:43:52,998 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0002 with negotiated timeout 40000 for
    client /10.40.222.77:59825
    2012-12-02 02:48:13,910 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from:
    /var/run/cloudera-scm-agent/process/34-zookeeper-server/zoo.cfg
    2012-12-02 02:48:13,914 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no
    config or no
    quorum defined in config, running in standalone mode
    2012-12-02 02:48:14,002 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from:
    /var/run/cloudera-scm-agent/process/34-zookeeper-server/zoo.cfg
    2012-12-02 02:48:14,003 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-02 02:48:14,030 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-02 02:48:14,030 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-02 02:48:14,030 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/34-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/34-zookeeper-server
    2012-12-02 02:48:14,039 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-02 02:48:14,039 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-02 02:48:14,040 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-02 02:48:14,072 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-02 02:48:14,088 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.28
    2012-12-02 02:48:14,109 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog:
    Snapshotting: 2c
    2012-12-02 02:48:15,025 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:39458
    2012-12-02 02:48:15,029 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0002 at /
    10.40.222.77:39458
    2012-12-02 02:48:15,035 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0002 with negotiated timeout 40000 for
    client /10.40.222.77:39458
    2012-12-02 02:48:15,060 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:56157
    2012-12-02 02:48:15,062 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0000 at /
    10.83.35.173:56157
    2012-12-02 02:48:15,063 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0000 with negotiated timeout 40000 for
    client /10.83.35.173:56157
    2012-12-02 02:48:15,095 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:39459
    2012-12-02 02:48:15,095 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0001 at /
    10.40.222.77:39459
    2012-12-02 02:48:15,096 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0001 with negotiated timeout 40000 for
    client /10.40.222.77:39459
    2012-12-02 03:23:48,488 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.2d
    2012-12-02 03:23:48,560 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0001
    2012-12-02 03:23:48,570 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:39459 which had sessionid
    0x13b59e867fc0001
    2012-12-02 03:25:46,004 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54443
    2012-12-02 03:25:46,007 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54443
    2012-12-02 03:25:46,009 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30000 with negotiated timeout 40000 for
    client /10.40.222.77:54443
    2012-12-02 03:25:46,348 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30000, likely client has closed socket
    2012-12-02 03:25:46,349 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:54443 which had sessionid
    0x13b5a963ea30000
    2012-12-02 03:26:15,044 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b59e867fc0000, likely client has closed socket
    2012-12-02 03:26:15,045 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.83.35.173:56157 which had sessionid
    0x13b59e867fc0000
    2012-12-02 03:26:17,867 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b59e867fc0002, likely client has closed socket
    2012-12-02 03:26:17,867 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:39458 which had sessionid
    0x13b59e867fc0002
    2012-12-02 03:26:28,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30000, timeout of 40000ms exceeded
    2012-12-02 03:26:28,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30000
    2012-12-02 03:26:42,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b59e867fc0000, timeout of 40000ms exceeded
    2012-12-02 03:26:42,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0000
    2012-12-02 03:26:47,391 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54461
    2012-12-02 03:26:47,394 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54461
    2012-12-02 03:26:47,396 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30001 with negotiated timeout 40000 for
    client /10.40.222.77:54461
    2012-12-02 03:26:47,448 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54462
    2012-12-02 03:26:47,460 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54462
    2012-12-02 03:26:47,463 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30002 with negotiated timeout 40000 for
    client /10.40.222.77:54462
    2012-12-02 03:26:56,000 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b59e867fc0002, timeout of 40000ms exceeded
    2012-12-02 03:26:56,000 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0002
    2012-12-02 03:27:00,606 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:56794
    2012-12-02 03:27:00,614 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:56794
    2012-12-02 03:27:00,617 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30003 with negotiated timeout 40000 for
    client /10.83.35.173:56794
    2012-12-02 03:36:45,885 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from:
    /var/run/cloudera-scm-agent/process/43-zookeeper-server/zoo.cfg
    2012-12-02 03:36:45,889 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no
    config or no
    quorum defined in config, running in standalone mode
    2012-12-02 03:36:45,962 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from:
    /var/run/cloudera-scm-agent/process/43-zookeeper-server/zoo.cfg
    2012-12-02 03:36:45,962 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/43-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/43-zookeeper-server
    2012-12-02 03:36:45,985 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-02 03:36:45,985 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-02 03:36:45,985 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-02 03:36:46,024 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-02 03:36:46,042 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.2c
    2012-12-02 03:36:46,063 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog:
    Snapshotting: 36
    2012-12-02 03:36:46,531 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:37153
    2012-12-02 03:36:46,535 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30003 at /
    10.83.35.173:37153
    2012-12-02 03:36:46,539 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30003 with negotiated timeout 40000 for
    client /10.83.35.173:37153
    2012-12-02 03:36:46,883 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:47487
    2012-12-02 03:36:46,884 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30002 at /
    10.40.222.77:47487
    2012-12-02 03:36:46,884 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30002 with negotiated timeout 40000 for
    client /10.40.222.77:47487
    2012-12-02 03:36:47,310 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:47488
    2012-12-02 03:36:47,311 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30001 at /
    10.40.222.77:47488
    2012-12-02 03:36:47,312 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30001 with negotiated timeout 40000 for
    client /10.40.222.77:47488
    2012-12-02 03:41:46,616 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44776
    2012-12-02 03:41:46,619 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44776
    2012-12-02 03:41:46,620 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.37
    2012-12-02 03:41:46,674 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0000 with negotiated timeout 40000 for
    client /10.40.222.77:44776
    2012-12-02 03:41:47,494 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5ac2ad7a0000, likely client has closed socket
    2012-12-02 03:41:47,495 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:44776 which had sessionid
    0x13b5ac2ad7a0000
    2012-12-02 03:42:16,001 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30003, likely client has closed socket
    2012-12-02 03:42:16,002 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.83.35.173:37153 which had sessionid
    0x13b5a963ea30003
    2012-12-02 03:42:17,916 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30002, likely client has closed socket
    2012-12-02 03:42:17,917 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:47487 which had sessionid
    0x13b5a963ea30002
    2012-12-02 03:42:19,468 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30001, likely client has closed socket
    2012-12-02 03:42:19,469 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:47488 which had sessionid
    0x13b5a963ea30001
    2012-12-02 03:42:28,000 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5ac2ad7a0000, timeout of 40000ms exceeded
    2012-12-02 03:42:28,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5ac2ad7a0000
    2012-12-02 03:42:46,214 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:52486
    2012-12-02 03:42:46,217 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:52486
    2012-12-02 03:42:46,220 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0001 with negotiated timeout 40000 for
    client /10.83.35.173:52486
    2012-12-02 03:42:47,857 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44799
    2012-12-02 03:42:47,864 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44799
    2012-12-02 03:42:47,867 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0002 with negotiated timeout 40000 for
    client /10.40.222.77:44799

    2012-12-02 03:42:47,991 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
    KeeperException when processing sessionid:0x13b5ac2ad7a0002
    type:create
    cxid:0x6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
    Path:/hbase/master Error:KeeperErrorCode = NodeExists for
    /hbase/master
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30002, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30003, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30001, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30002
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30003
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30001
    2012-12-02 03:42:48,182 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44800
    2012-12-02 03:42:48,194 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44800
    2012-12-02 03:42:48,196 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0003 with negotiated timeout 40000 for
    client /10.40.222.77:44800
    Shouvanik,

    The errors look very much like
    http://zookeeper-user.578899.n2.nabble.com/Unusual-exception-td5632833.html

    Is your client on 10.40.222.77? It is connecting to ZK. Do you have
    any unusual network/firewall setting?

    Cheers,
    bc

    Regards,
    Shouvanik



    On Mon, Dec 3, 2012 at 1:44 AM, Harsh J <harsh@cloudera.com>
    wrote:
    Hi Shouvanik,

    Those logs are all INFO level and there's no issue in it. Can you
    share the issue/message you get when you mean "I am not able to
    start
    hbase" instead?

    On Mon, Dec 3, 2012 at 1:35 AM, Shouvanik Haldar
    wrote:
    Hi Harsh,

    I am not able to start hbase although hbase service is up. I am
    getting
    weird zookeeper exception. can you please help

    [root@ip-10-40-222-77 ~]# tail -f
    /var/log/zookeeper/zookeeper-cmf-zookeeper1-SERVER-ip-10-40-222-77.ec2.internal.log
    2012-12-02 03:42:47,991 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
    KeeperException when processing sessionid:0x13b5ac2ad7a0002
    type:create
    cxid:0x6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
    Error
    Path:/hbase/master Error:KeeperErrorCode = NodeExists for
    /hbase/master
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30002, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30003, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30001, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed
    session
    termination for sessionid: 0x13b5a963ea30002
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed
    session
    termination for sessionid: 0x13b5a963ea30003
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed
    session
    termination for sessionid: 0x13b5a963ea30001
    2012-12-02 03:42:48,182 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44800
    2012-12-02 03:42:48,194 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /
    10.40.222.77:44800
    2012-12-02 03:42:48,196 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0003 with negotiated timeout
    40000 for
    client /10.40.222.77:44800


    Regards,
    Shouvanik Haldar



    On Mon, Dec 3, 2012 at 1:24 AM, Harsh J <harsh@cloudera.com>
    wrote:
    I replied to this over your duplicate post at

    https://groups.google.com/a/cloudera.org/d/topic/cdh-user/ft02DH52BZ4/discussion
    On Mon, Dec 3, 2012 at 12:59 AM, John H <john.hadoop@gmail.com>
    wrote:
    Hi,

    I installed Cloudera Manager(4.1.1) and CDH4 (4.1.2) using
    tarball
    on
    Ubuntu 10.04.4 (VM using virtual box). I was able to setup
    Cloudera
    manager
    server and agent, setup roles (NN,DN and Secondary NN) in
    pseudo
    distributed
    mode. I deployed the configuration to hosts and formatted
    name node
    successfully.

    When I started the services from Cloudera manager, I got
    message
    saying
    all
    3 (NN,DN,SNN) were started successfully, but I cannot access
    anyone
    of
    them
    through web url.
    The status for all 3 is started with good health.


    I looked at the log file for namenode and it is stuck at
    this line


    "2012-12-02 10:33:33,841 INFO
    org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager:
    Going
    to
    retain 2 images with txid >= 0"

    This is the first start after format, so I am confused why
    this is
    happening... Any help would be really appreciated..


    Thanks
    JH





    FULL LOG FILE: ***************************



    ************************************************************/
    2012-12-02 10:32:19,931 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting NameNode
    STARTUP_MSG: host = ubuntu1/127.0.0.1
    STARTUP_MSG: args = []
    STARTUP_MSG: version = 2.0.0-cdh4.1.2
    STARTUP_MSG: classpath =

    /home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-log4j12-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-json-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jettison-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jets3t-0.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-digester-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-annotations-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/activation-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/kfs-0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-net-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-auth-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsch-0.1.42.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-api-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/paranamer-2.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-math-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/junit-4.8.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/avro-1.7.1.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/stax-api-1.0.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-tests.jar:/home/john/cloudera-manager/cm-4.1.1/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-daemon-1.0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-tests.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-sources.jar:/usr/lib/hadoop-yarn/share/hadoop/yarn/*:/usr/lib/hadoop-mapreduce/share/hadoop/mapreduce/*
    STARTUP_MSG: build =

    file:///var/lib/jenkins/workspace/CDH4.1.2-Packaging-Hadoop/build/cdh4/hadoop/2.0.0-cdh4.1.2/source/hadoop-common-project/hadoop-common
    -r f0b53c81cbf56f5955e403b49fcd27afd5f082de; compiled by
    'jenkins' on
    Thu
    Nov 1 17:06:55 PDT 2012
    ************************************************************/
    2012-12-02 10:32:22,214 INFO
    org.apache.hadoop.metrics2.impl.MetricsConfig:
    loaded properties from hadoop-metrics2.properties
    2012-12-02 10:32:23,240 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled
    snapshot
    period
    at 10 second(s).
    2012-12-02 10:32:23,240 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode
    metrics
    system
    started
    2012-12-02 10:32:25,269 INFO
    org.apache.hadoop.util.HostsFileReader:
    Refreshing hosts (include/exclude) list
    2012-12-02 10:32:25,270 INFO
    org.apache.hadoop.util.HostsFileReader:
    Adding
    127.0.0.1 to the list of hosts from

    /home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE/dfs_hosts_allow.txt
    2012-12-02 10:32:25,362 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
    dfs.block.invalidate.limit=1000
    2012-12-02 10:32:25,422 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    dfs.block.access.token.enable=false
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    defaultReplication = 1
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    maxReplication
    = 512
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    minReplication
    = 1
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    maxReplicationStreams = 2
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    shouldCheckForEnoughRacks = true
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    replicationRecheckInterval = 3000
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    encryptDataTransfer = false
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
    =
    john (auth:SIMPLE)
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    supergroup
    =
    john
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    isPermissionEnabled
    =
    true
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA
    Enabled:
    false
    2012-12-02 10:32:25,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append
    Enabled:
    true
    2012-12-02 10:32:27,741 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: Caching
    file names
    occuring
    more than 10 times
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.threshold-pct = 0.9990000128746033
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.min.datanodes = 0
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.extension = 30000
    2012-12-02 10:32:27,941 INFO
    org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /home/john/dfs_dir/namenode/data1/in_use.lock
    acquired by
    nodename
    2070@ubuntu1
    2012-12-02 10:32:28,574 INFO
    org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /home/john/dfs_dir/namenode/data2/in_use.lock
    acquired by
    nodename
    2070@ubuntu1
    2012-12-02 10:32:28,803 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Recovering
    unfinalized segments in
    /home/john/dfs_dir/namenode/data1/current
    2012-12-02 10:32:28,803 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Recovering
    unfinalized segments in
    /home/john/dfs_dir/namenode/data2/current
    2012-12-02 10:32:28,805 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    No edit log streams selected.
    2012-12-02 10:32:28,950 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Loading image file
    /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000
    using
    no compression
    2012-12-02 10:32:28,950 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Number of files = 1
    2012-12-02 10:32:28,951 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Number of files under construction = 0
    2012-12-02 10:32:28,952 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Image file of size 117 loaded in 0 seconds.
    2012-12-02 10:32:28,952 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Loaded image for txid 0 from
    /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000
    2012-12-02 10:32:28,955 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting
    log
    segment
    at 1
    2012-12-02 10:32:30,041 INFO
    org.apache.hadoop.hdfs.server.namenode.NameCache:
    initialized with 0
    entries
    0 lookups
    2012-12-02 10:32:30,041 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    Finished loading
    FSImage in 2298 msecs
    2012-12-02 10:32:31,360 INFO org.apache.hadoop.ipc.Server:
    Starting
    Socket
    Reader #1 for port 8020
    2012-12-02 10:32:31,402 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    Registered
    FSNamesystemState MBean
    2012-12-02 10:32:31,482 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number
    of blocks
    under
    construction: 0
    2012-12-02 10:32:31,482 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    initializing
    replication queues
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Total
    number
    of
    blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number of
    invalid blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number of
    under-replicated blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number of
    over-replicated blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number of
    blocks
    being written = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.StateChange:
    STATE*
    Replication Queue initialization scan for invalid, over- and
    under-replicated blocks completed in 31 msec
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.StateChange:
    STATE*
    Leaving safe mode after 6 secs.
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.StateChange:
    STATE*
    Network topology has 0 racks and 0 datanodes
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.StateChange:
    STATE*
    UnderReplicatedBlocks has 0 blocks
    2012-12-02 10:32:31,674 INFO org.mortbay.log: Logging to
    org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
    org.mortbay.log.Slf4jLog
    2012-12-02 10:32:31,814 INFO
    org.apache.hadoop.http.HttpServer: Added
    global
    filter 'safety'
    (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    2012-12-02 10:32:31,815 INFO
    org.apache.hadoop.http.HttpServer: Added
    filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context hdfs
    2012-12-02 10:32:31,815 INFO
    org.apache.hadoop.http.HttpServer: Added
    filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context logs
    2012-12-02 10:32:31,815 INFO
    org.apache.hadoop.http.HttpServer: Added
    filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context static
    2012-12-02 10:32:31,841 INFO
    org.apache.hadoop.http.HttpServer:
    dfs.webhdfs.enabled = true
    2012-12-02 10:32:31,842 INFO
    org.apache.hadoop.http.HttpServer: Added
    filter
    'SPNEGO' (class=org.apache.hadoop.hdfs.web.AuthFilter)
    2012-12-02 10:32:31,844 INFO
    org.apache.hadoop.http.HttpServer:
    addJerseyResourcePackage:

    packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
    pathSpec=/webhdfs/v1/*
    2012-12-02 10:32:31,862 INFO
    org.apache.hadoop.http.HttpServer: Jetty
    bound
    to port 50070
    2012-12-02 10:32:31,862 INFO org.mortbay.log:
    jetty-6.1.26.cloudera.2
    2012-12-02 10:32:32,261 WARN
    org.apache.hadoop.security.authentication.server.AuthenticationFilter:
    'signature.secret' configuration not set, using a random
    value as
    secret
    2012-12-02 10:32:32,453 INFO org.mortbay.log: Started
    selectchannelconnector@localhost.localdomain:50070
    2012-12-02 10:32:32,453 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server
    up at:
    localhost.localdomain:50070
    2012-12-02 10:32:32,453 INFO org.apache.hadoop.ipc.Server:
    IPC Server
    Responder: starting
    2012-12-02 10:32:32,454 INFO org.apache.hadoop.ipc.Server:
    IPC Server
    listener on 8020: starting
    2012-12-02 10:32:32,603 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode up
    at:
    ubuntu1/127.0.0.1:8020
    2012-12-02 10:32:32,603 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting
    services
    required for active state
    2012-12-02 10:32:34,359 INFO
    org.apache.hadoop.fs.TrashPolicyDefault:
    The
    configured checkpoint interval is 0 minutes. Using an
    interval of
    1440
    minutes that is used for deletion instead
    2012-12-02 10:32:35,780 INFO
    org.apache.hadoop.hdfs.StateChange:
    BLOCK*
    NameSystem.registerDatanode: node registration from
    DatanodeRegistration(127.0.0.1,
    storageID=DS-377461142-127.0.0.1-50010-1354473155123,
    infoPort=50075,
    ipcPort=50020,
    storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0)
    storage
    DS-377461142-127.0.0.1-50010-1354473155123
    2012-12-02 10:32:35,790 INFO
    org.apache.hadoop.net.NetworkTopology:
    Adding a
    new node: /default/127.0.0.1:50010
    2012-12-02 10:32:35,951 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    BLOCK*
    processReport: Received first block report from
    127.0.0.1:50010 after
    becoming active. Its block contents are no longer considered
    stale.
    2012-12-02 10:32:35,951 INFO
    org.apache.hadoop.hdfs.StateChange:
    BLOCK*
    processReport: from DatanodeRegistration(127.0.0.1,
    storageID=DS-377461142-127.0.0.1-50010-1354473155123,
    infoPort=50075,
    ipcPort=50020,
    storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0),
    blocks:
    0, processing time: 1 msecs
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll
    Edit Log
    from
    127.0.0.1
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling
    edit logs.
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
    segment
    1
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
    transactions: 2
    Total time for transactions(ms): 1Number of transactions
    batched in
    Syncs: 0
    Number of syncs: 2 SyncTimes(ms): 166 189
    2012-12-02 10:33:31,595 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
    transactions: 2
    Total time for transactions(ms): 1Number of transactions
    batched in
    Syncs: 0
    Number of syncs: 3 SyncTimes(ms): 215 255
    2012-12-02 10:33:31,603 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Finalizing
    edits
    file

    /home/john/dfs_dir/namenode/data1/current/edits_inprogress_0000000000000000001
    ->

    /home/john/dfs_dir/namenode/data1/current/edits_0000000000000000001-0000000000000000002
    2012-12-02 10:33:31,604 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Finalizing
    edits
    file

    /home/john/dfs_dir/namenode/data2/current/edits_inprogress_0000000000000000001
    ->

    /home/john/dfs_dir/namenode/data2/current/edits_0000000000000000001-0000000000000000002
    2012-12-02 10:33:31,604 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting
    log
    segment
    at 3
    2012-12-02 10:33:33,274 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage:
    Opening
    connection
    to

    http://ubuntu1:50090/getimage?getimage=1&txid=2&storageInfo=-40:829593698:0:cluster2
    2012-12-02 10:33:33,622 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage:
    Transfer took
    0.34s
    at 0.00 KB/s
    2012-12-02 10:33:33,622 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage:
    Downloaded
    file
    fsimage.ckpt_0000000000000000002 size 117 bytes.
    2012-12-02 10:33:33,841 INFO
    org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager:
    Going
    to
    retain 2 images with txid >= 0



    --
    Harsh J



    --
    Thanks,
    Shouvanik


    --
    Harsh J



    --
    Thanks,
    Shouvanik


    --
    Harsh J

    --
    Thanks,
    *Shouvanik*

    --
    Thanks,
    *Shouvanik*

    --
    Thanks,
    *Shouvanik*
  • Shouvanik Haldar at Dec 4, 2012 at 6:31 pm
    Yes. I can do a 'hadoop fs -touchz /tmp/foo' without any error.
    By HDFS daemons, do you mean logs that are inside /var/log/hadoop folder?

    If not, can you please tell me where to search for them?


    *In the log file
    "/var/log/hadoop/hadoop-cmf-mapreduce1-JOBTRACKER-ip-10-40-222-77.ec2.internal.log.out",
    I find the below logs which might be important for you to find out the
    cause.*


    2012-12-04 06:43:28,231 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
    Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
    /tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes,
    instead of 1

    2012-12-04 06:43:28,231 WARN org.apache.hadoop.hdfs.DFSClient: Could not
    get block locations. Source file "/tmp/mapred/system/jobtracker.info" -
    Aborting...
    2012-12-04 06:43:28,231 WARN org.apache.hadoop.mapred.JobTracker: Writing
    to file hdfs://ip-10-40-222-77.ec2.internal/tmp/mapred/system/
    jobtracker.info failed!


    2012-12-04 06:43:38,257 WARN org.apache.hadoop.hdfs.DFSClient: Error
    Recovery for block null bad datanode[0] nodes == null
    2012-12-04 06:43:38,257 WARN org.apache.hadoop.hdfs.DFSClient: Could not
    get block locations. Source file "/tmp/mapred/system/jobtracker.info" -
    Aborting...
    2012-12-04 06:43:38,257 WARN org.apache.hadoop.mapred.JobTracker: Writing
    to file hdfs://ip-10-40-222-77.ec2.internal/tmp/mapred/system/
    jobtracker.info failed!


    *And below are logs from
    "/var/log/hadoop/hadoop-cmf-hdfs1-DATANODE-ip-10-40-222-77.ec2.internal.log.out"
    *


    2012-12-04 07:01:14,131 WARN
    org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Call
    to ip-10-40-222-77.ec2.internal/10.40.222.77:8020 failed on local
    exception: java.io.EOFException

    Please help!

    Regards,
    Shouvanik



    On Tue, Dec 4, 2012 at 11:25 PM, bc Wong wrote:

    On Tue, Dec 4, 2012 at 9:02 AM, Shouvanik Haldar <
    shouvanik.haldar@gmail.com> wrote:
    Hi,

    I did that and below is the list of client connections

    tcp 0 0 ::ffff:10.40.222.77:2181 ::ffff:10.40.222.77:43512
    ESTABLISHED 20441/java
    tcp 0 0 ::ffff:10.40.222.77:2181 ::ffff:10.40.222.77:43513
    ESTABLISHED 20441/java
    tcp 0 0 ::ffff:10.40.222.77:43512 ::ffff:10.40.222.77:2181
    ESTABLISHED 22681/java
    tcp 0 0 ::ffff:10.40.222.77:43513 ::ffff:10.40.222.77:2181
    ESTABLISHED 22707/java
    tcp 0 0 ::ffff:10.40.222.77:51510 ::ffff:10.40.222.77:2181
    TIME_WAIT -

    Is that a problem?
    Looks ok.

    Please help me folks...I am not able to execute simple hbase query..since
    I am not able to connect to HBase...below I paste all logs that I think may
    give some hint why I am not able to achieve success




    2012-12-04 06:34:56,057 ERROR
    org.apache.hadoop.hbase.master.HMasterCommandLine: Master not running

    ERROR org.apache.hadoop.hdfs.DFSClient: Exception closing file
    /hbase/hbase.version : org.apache.hadoop.ipc.RemoteException:
    java.io.IOException: File /hbase/hbase.version could only be replicated to
    0 nodes, instead of 1
    This error is new (for your cluster). Can you go to your HDFS daemons and
    look for errors in the logs? Can you successfully do a simple `hadoop fs
    -touchz /tmp/foo'?

    Cheers,
    bc

    2012-12-04 09:56:33,009 INFO
    org.apache.hadoop.hbase.master.ActiveMasterManager: Master=
    ec2-75-101-165-36.compute-1.amazonaws.com:60000
    2012-12-04 09:56:33,438 WARN org.apache.hadoop.hdfs.DFSClient:
    DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
    java.io.IOException: File /hbase/hbase.version could only be replicated to
    0 nodes, instead of 1


    2012-12-04 09:56:43,450 WARN org.apache.hadoop.hdfs.DFSClient:
    DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
    java.io.IOException: File /hbase/hbase.version could only be replicated to
    0 nodes, instead of 1

    2012-12-04 09:57:43,533 WARN org.apache.hadoop.hdfs.DFSClient: Error
    Recovery for block null bad datanode[0] nodes == null





    12/12/04 11:13:06 ERROR zookeeper.ZKConfig: no valid quorum servers found
    in zoo.cfg

    ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: An error is
    preventing HBase from connecting to ZooKeeper

    2012-12-04 11:19:06,584 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b66b4d47c0000, likely client has closed socket

    2012-12-04 11:20:14,000 INFO org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b65bbe40c0013, timeout of 40000ms exceeded

    Got user-level KeeperException when processing
    sessionid:0x13b66b4d47c0001 type:create cxid:0x6 zxid:0xfffffffffffffffe
    txntype:unknown reqpath:n/a Error Path:/hbase/master Error:KeeperErrorCode
    = NodeExists for /hbase/master



    ERROR: org.apache.hadoop.hbase.MasterNotRunningException:
    ec2-75-101-165-36.compute-1.amazonaws.com:60000



    On Tue, Dec 4, 2012 at 12:58 AM, bc Wong wrote:

    On Mon, Dec 3, 2012 at 1:40 AM, Shouvanik Haldar <
    shouvanik.haldar@gmail.com> wrote:
    Hi,

    I performed the steps as you told me. Still I am facing the previously
    pasted ZK exception.
    Where can I find the zookeeper logs that cloudera manager writes to? In
    the admin console, I can find the log location as /var/log/zookeeper. Is
    this the only location?

    It's in /var/log/zookeeper. Can you run `netstat -ntp | grep 2181' on
    your ZK node to see whether there are a lot of client connections?

    Cheers,
    bc

    On Mon, Dec 3, 2012 at 1:42 PM, bc Wong wrote:

    On Sun, Dec 2, 2012 at 11:18 PM, Shouvanik Haldar <
    shouvanik.haldar@gmail.com> wrote:
    Hi,

    Cloudera and all the services require some ports. These are enabled
    in AWS security group. Except that I am not aware of any firewall settings.

    Tell me one more thing. Does the /etc/hosts have anything to do?

    Let me paste the contents of this file.

    *127.0.0.1 localhost
    10.40.222.77 ip-10-40-222-77*


    and for the other server

    *127.0.0.1 localhost
    10.83.35.173 ip-10-83-35-173*

    Is this OK?
    Looks fine. Maybe you should heed this warning:

    "ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase
    is able
    to connect to ZooKeeper but the connection closes immediately. This
    could be
    a sign that the server has too many connections (30 is the default).
    Consider inspecting your ZK server logs for that error and then make
    sure
    you are reusing HBaseConfiguration as often as you can. See HTable's
    javadoc
    for more information."

    Can you restart your ZK and HBase services, and them immediately try
    to do a "describe".

    Another help.
    Can you please point to the exact guide to upgrade to CM 4.x?

    https://ccp.cloudera.com/display/ENT4DOC/Upgrading+to+Cloudera+Manager+4.0

    See the subsections on upgrading from/to different versions.

    Cheers,
    bc

    Regards,
    Shouvanik



    On Mon, Dec 3, 2012 at 12:15 PM, bc Wong wrote:
    On Sun, Dec 2, 2012 at 12:34 PM, Harsh J wrote:

    Hi Shouvanik,

    I notice you use CDH3. You should really be using CDH4, as it
    provides
    a great deal of improvements over the CDH3 version that you'll be
    missing out on.

    In any case:

    If there exists a /etc/zookeeper/zoo.cfg on your client machine,
    please rename it to something else, such as zoo.cfg.unwanted, or
    remove it altogether, and retry.

    Alternatively, try to run your HBase shell command(s) with the
    direct
    script (instead of wrapper /usr/bin/hbase) as:

    /usr/lib/hbase/bin/hbase shell

    And see if it lets you connect and work normally this time.

    On Mon, Dec 3, 2012 at 1:50 AM, Shouvanik Haldar
    wrote:
    Yes Harsh,

    [root@ip-10-40-222-77 ~]# hbase shell
    HBase Shell; enter 'help<RETURN>' for list of supported commands.
    Type "exit<RETURN>" to leave the HBase Shell
    Version 0.90.4-cdh3u3, r, Thu Jan 26 10:13:36 PST 2012

    hbase(main):001:0> describe 't1'

    ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException:
    HBase is able
    to connect to ZooKeeper but the connection closes immediately.
    This could be
    a sign that the server has too many connections (30 is the default).
    Consider inspecting your ZK server logs for that error and then make sure
    you are reusing HBaseConfiguration as often as you can. See
    HTable's javadoc
    for more information.

    Here is some help for this command:
    Describe the named table. For example:
    hbase> describe 't1'


    hbase(main):002:0>


    Secondly,

    When I see the zookeeper logs, it says

    [root@ip-10-40-222-77 ~]# cat
    /var/log/zookeeper/zookeeper-cmf-zookeeper1-SERVER-ip-10-40-222-77.ec2.internal.log
    2012-12-01 23:30:35,726 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from:
    /var/run/cloudera-scm-agent/process/1-zookeeper-server/zoo.cfg
    2012-12-01 23:30:35,731 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no
    config or no
    quorum defined in config, running in standalone mode
    2012-12-01 23:30:35,811 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from:
    /var/run/cloudera-scm-agent/process/1-zookeeper-server/zoo.cfg
    2012-12-01 23:30:35,812 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-01 23:30:35,842 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/1-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/1-zookeeper-server
    2012-12-01 23:30:35,851 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-01 23:30:35,851 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-01 23:30:35,851 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-01 23:30:35,891 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-01 23:30:35,907 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.0
    2012-12-01 23:30:35,941 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog:
    Snapshotting: 28
    2012-12-01 23:38:21,260 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from:
    /var/run/cloudera-scm-agent/process/9-zookeeper-server/zoo.cfg
    2012-12-01 23:38:21,265 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no
    config or no
    quorum defined in config, running in standalone mode
    2012-12-01 23:38:21,339 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from:
    /var/run/cloudera-scm-agent/process/9-zookeeper-server/zoo.cfg
    2012-12-01 23:38:21,340 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/9-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/9-zookeeper-server
    2012-12-01 23:38:21,366 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-01 23:38:21,366 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-01 23:38:21,366 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-01 23:38:21,399 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-01 23:38:21,415 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.28
    2012-12-01 23:38:21,430 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog:
    Snapshotting: 28
    2012-12-01 23:43:49,121 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:45741
    2012-12-01 23:43:49,128 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:45741
    2012-12-01 23:43:49,130 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.29
    2012-12-01 23:43:49,172 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0000 with negotiated timeout 40000 for
    client /10.83.35.173:45741
    2012-12-01 23:43:52,894 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:59824
    2012-12-01 23:43:52,897 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:59824
    2012-12-01 23:43:52,899 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0001 with negotiated timeout 40000 for
    client /10.40.222.77:59824
    2012-12-01 23:43:52,993 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:59825
    2012-12-01 23:43:52,996 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:59825
    2012-12-01 23:43:52,998 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0002 with negotiated timeout 40000 for
    client /10.40.222.77:59825
    2012-12-02 02:48:13,910 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from:
    /var/run/cloudera-scm-agent/process/34-zookeeper-server/zoo.cfg
    2012-12-02 02:48:13,914 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no
    config or no
    quorum defined in config, running in standalone mode
    2012-12-02 02:48:14,002 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from:
    /var/run/cloudera-scm-agent/process/34-zookeeper-server/zoo.cfg
    2012-12-02 02:48:14,003 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-02 02:48:14,030 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-02 02:48:14,030 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-02 02:48:14,030 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/34-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/34-zookeeper-server
    2012-12-02 02:48:14,039 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-02 02:48:14,039 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-02 02:48:14,040 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-02 02:48:14,072 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-02 02:48:14,088 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.28
    2012-12-02 02:48:14,109 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog:
    Snapshotting: 2c
    2012-12-02 02:48:15,025 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:39458
    2012-12-02 02:48:15,029 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0002 at /
    10.40.222.77:39458
    2012-12-02 02:48:15,035 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0002 with negotiated timeout 40000 for
    client /10.40.222.77:39458
    2012-12-02 02:48:15,060 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:56157
    2012-12-02 02:48:15,062 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0000 at /
    10.83.35.173:56157
    2012-12-02 02:48:15,063 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0000 with negotiated timeout 40000 for
    client /10.83.35.173:56157
    2012-12-02 02:48:15,095 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:39459
    2012-12-02 02:48:15,095 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0001 at /
    10.40.222.77:39459
    2012-12-02 02:48:15,096 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0001 with negotiated timeout 40000 for
    client /10.40.222.77:39459
    2012-12-02 03:23:48,488 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.2d
    2012-12-02 03:23:48,560 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0001
    2012-12-02 03:23:48,570 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:39459 which
    had sessionid
    0x13b59e867fc0001
    2012-12-02 03:25:46,004 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54443
    2012-12-02 03:25:46,007 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54443
    2012-12-02 03:25:46,009 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30000 with negotiated timeout 40000 for
    client /10.40.222.77:54443
    2012-12-02 03:25:46,348 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30000, likely client has closed socket
    2012-12-02 03:25:46,349 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:54443 which
    had sessionid
    0x13b5a963ea30000
    2012-12-02 03:26:15,044 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b59e867fc0000, likely client has closed socket
    2012-12-02 03:26:15,045 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.83.35.173:56157 which
    had sessionid
    0x13b59e867fc0000
    2012-12-02 03:26:17,867 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b59e867fc0002, likely client has closed socket
    2012-12-02 03:26:17,867 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:39458 which
    had sessionid
    0x13b59e867fc0002
    2012-12-02 03:26:28,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30000, timeout of 40000ms exceeded
    2012-12-02 03:26:28,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30000
    2012-12-02 03:26:42,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b59e867fc0000, timeout of 40000ms exceeded
    2012-12-02 03:26:42,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0000
    2012-12-02 03:26:47,391 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54461
    2012-12-02 03:26:47,394 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54461
    2012-12-02 03:26:47,396 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30001 with negotiated timeout 40000 for
    client /10.40.222.77:54461
    2012-12-02 03:26:47,448 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54462
    2012-12-02 03:26:47,460 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:54462
    2012-12-02 03:26:47,463 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30002 with negotiated timeout 40000 for
    client /10.40.222.77:54462
    2012-12-02 03:26:56,000 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b59e867fc0002, timeout of 40000ms exceeded
    2012-12-02 03:26:56,000 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0002
    2012-12-02 03:27:00,606 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:56794
    2012-12-02 03:27:00,614 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:56794
    2012-12-02 03:27:00,617 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30003 with negotiated timeout 40000 for
    client /10.83.35.173:56794
    2012-12-02 03:36:45,885 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from:
    /var/run/cloudera-scm-agent/process/43-zookeeper-server/zoo.cfg
    2012-12-02 03:36:45,889 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no
    config or no
    quorum defined in config, running in standalone mode
    2012-12-02 03:36:45,962 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from:
    /var/run/cloudera-scm-agent/process/43-zookeeper-server/zoo.cfg
    2012-12-02 03:36:45,962 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/43-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/43-zookeeper-server
    2012-12-02 03:36:45,985 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-02 03:36:45,985 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-02 03:36:45,985 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-02 03:36:46,024 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-02 03:36:46,042 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.2c
    2012-12-02 03:36:46,063 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog:
    Snapshotting: 36
    2012-12-02 03:36:46,531 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:37153
    2012-12-02 03:36:46,535 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30003 at /
    10.83.35.173:37153
    2012-12-02 03:36:46,539 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30003 with negotiated timeout 40000 for
    client /10.83.35.173:37153
    2012-12-02 03:36:46,883 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:47487
    2012-12-02 03:36:46,884 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30002 at /
    10.40.222.77:47487
    2012-12-02 03:36:46,884 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30002 with negotiated timeout 40000 for
    client /10.40.222.77:47487
    2012-12-02 03:36:47,310 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:47488
    2012-12-02 03:36:47,311 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30001 at /
    10.40.222.77:47488
    2012-12-02 03:36:47,312 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30001 with negotiated timeout 40000 for
    client /10.40.222.77:47488
    2012-12-02 03:41:46,616 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44776
    2012-12-02 03:41:46,619 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44776
    2012-12-02 03:41:46,620 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.37
    2012-12-02 03:41:46,674 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0000 with negotiated timeout 40000 for
    client /10.40.222.77:44776
    2012-12-02 03:41:47,494 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5ac2ad7a0000, likely client has closed socket
    2012-12-02 03:41:47,495 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:44776 which
    had sessionid
    0x13b5ac2ad7a0000
    2012-12-02 03:42:16,001 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30003, likely client has closed socket
    2012-12-02 03:42:16,002 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.83.35.173:37153 which
    had sessionid
    0x13b5a963ea30003
    2012-12-02 03:42:17,916 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30002, likely client has closed socket
    2012-12-02 03:42:17,917 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:47487 which
    had sessionid
    0x13b5a963ea30002
    2012-12-02 03:42:19,468 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30001, likely client has closed socket
    2012-12-02 03:42:19,469 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:47488 which
    had sessionid
    0x13b5a963ea30001
    2012-12-02 03:42:28,000 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5ac2ad7a0000, timeout of 40000ms exceeded
    2012-12-02 03:42:28,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5ac2ad7a0000
    2012-12-02 03:42:46,214 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:52486
    2012-12-02 03:42:46,217 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.83.35.173:52486
    2012-12-02 03:42:46,220 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0001 with negotiated timeout 40000 for
    client /10.83.35.173:52486
    2012-12-02 03:42:47,857 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44799
    2012-12-02 03:42:47,864 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44799
    2012-12-02 03:42:47,867 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0002 with negotiated timeout 40000 for
    client /10.40.222.77:44799

    2012-12-02 03:42:47,991 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
    KeeperException when processing sessionid:0x13b5ac2ad7a0002
    type:create
    cxid:0x6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
    Path:/hbase/master Error:KeeperErrorCode = NodeExists for
    /hbase/master
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30002, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30003, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30001, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30002
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30003
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30001
    2012-12-02 03:42:48,182 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44800
    2012-12-02 03:42:48,194 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /10.40.222.77:44800
    2012-12-02 03:42:48,196 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0003 with negotiated timeout 40000 for
    client /10.40.222.77:44800
    Shouvanik,

    The errors look very much like
    http://zookeeper-user.578899.n2.nabble.com/Unusual-exception-td5632833.html

    Is your client on 10.40.222.77? It is connecting to ZK. Do you have
    any unusual network/firewall setting?

    Cheers,
    bc

    Regards,
    Shouvanik



    On Mon, Dec 3, 2012 at 1:44 AM, Harsh J <harsh@cloudera.com>
    wrote:
    Hi Shouvanik,

    Those logs are all INFO level and there's no issue in it. Can you
    share the issue/message you get when you mean "I am not able to
    start
    hbase" instead?

    On Mon, Dec 3, 2012 at 1:35 AM, Shouvanik Haldar
    wrote:
    Hi Harsh,

    I am not able to start hbase although hbase service is up. I
    am getting
    weird zookeeper exception. can you please help

    [root@ip-10-40-222-77 ~]# tail -f
    /var/log/zookeeper/zookeeper-cmf-zookeeper1-SERVER-ip-10-40-222-77.ec2.internal.log
    2012-12-02 03:42:47,991 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Got
    user-level
    KeeperException when processing sessionid:0x13b5ac2ad7a0002
    type:create
    cxid:0x6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
    Error
    Path:/hbase/master Error:KeeperErrorCode = NodeExists for
    /hbase/master
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30002, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30003, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30001, timeout of 40000ms exceeded
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed
    session
    termination for sessionid: 0x13b5a963ea30002
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed
    session
    termination for sessionid: 0x13b5a963ea30003
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed
    session
    termination for sessionid: 0x13b5a963ea30001
    2012-12-02 03:42:48,182 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44800
    2012-12-02 03:42:48,194 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /
    10.40.222.77:44800
    2012-12-02 03:42:48,196 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0003 with negotiated timeout
    40000 for
    client /10.40.222.77:44800


    Regards,
    Shouvanik Haldar



    On Mon, Dec 3, 2012 at 1:24 AM, Harsh J <harsh@cloudera.com>
    wrote:
    I replied to this over your duplicate post at

    https://groups.google.com/a/cloudera.org/d/topic/cdh-user/ft02DH52BZ4/discussion
    On Mon, Dec 3, 2012 at 12:59 AM, John H <
    john.hadoop@gmail.com> wrote:
    Hi,

    I installed Cloudera Manager(4.1.1) and CDH4 (4.1.2)
    using tarball
    on
    Ubuntu 10.04.4 (VM using virtual box). I was able to setup
    Cloudera
    manager
    server and agent, setup roles (NN,DN and Secondary NN) in
    pseudo
    distributed
    mode. I deployed the configuration to hosts and formatted
    name node
    successfully.

    When I started the services from Cloudera manager, I got
    message
    saying
    all
    3 (NN,DN,SNN) were started successfully, but I cannot
    access anyone
    of
    them
    through web url.
    The status for all 3 is started with good health.


    I looked at the log file for namenode and it is stuck at
    this line


    "2012-12-02 10:33:33,841 INFO
    org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager:
    Going
    to
    retain 2 images with txid >= 0"

    This is the first start after format, so I am confused why
    this is
    happening... Any help would be really appreciated..


    Thanks
    JH





    FULL LOG FILE: ***************************


    ************************************************************/
    2012-12-02 10:32:19,931 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode:
    STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting NameNode
    STARTUP_MSG: host = ubuntu1/127.0.0.1
    STARTUP_MSG: args = []
    STARTUP_MSG: version = 2.0.0-cdh4.1.2
    STARTUP_MSG: classpath =

    /home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-log4j12-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-json-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jettison-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jets3t-0.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-digester-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-annotations-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/activation-1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/kfs-0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-net-3.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/hadoop-auth-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsch-0.1.42.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/slf4j-api-1.6.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/paranamer-2.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-math-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/junit-4.8.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/avro-1.7.1.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/stax-api-1.0.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/common/hadoop-common-2.0.0-cdh4.1.2-tests.jar:/home/john/cloudera-manager/cm-4.1.1/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/protobuf-java-2.4.0a.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-io-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-core-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/asm-3.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jersey-server-1.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-lang-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jetty-6.1.26.cloudera.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/zookeeper-3.4.3-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jline-0.9.94.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-logging-1.1.1.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-daemon-1.0.3.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-test-sources.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-tests.jar:/home/john/hdfs/hadoop-2.0.0-cdh4.1.2/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.1.2-sources.jar:/usr/lib/hadoop-yarn/share/hadoop/yarn/*:/usr/lib/hadoop-mapreduce/share/hadoop/mapreduce/*
    STARTUP_MSG: build =

    file:///var/lib/jenkins/workspace/CDH4.1.2-Packaging-Hadoop/build/cdh4/hadoop/2.0.0-cdh4.1.2/source/hadoop-common-project/hadoop-common
    -r f0b53c81cbf56f5955e403b49fcd27afd5f082de; compiled by
    'jenkins' on
    Thu
    Nov 1 17:06:55 PDT 2012
    ************************************************************/
    2012-12-02 10:32:22,214 INFO
    org.apache.hadoop.metrics2.impl.MetricsConfig:
    loaded properties from hadoop-metrics2.properties
    2012-12-02 10:32:23,240 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
    Scheduled snapshot
    period
    at 10 second(s).
    2012-12-02 10:32:23,240 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode
    metrics
    system
    started
    2012-12-02 10:32:25,269 INFO
    org.apache.hadoop.util.HostsFileReader:
    Refreshing hosts (include/exclude) list
    2012-12-02 10:32:25,270 INFO
    org.apache.hadoop.util.HostsFileReader:
    Adding
    127.0.0.1 to the list of hosts from

    /home/john/cloudera-manager/cm-4.1.1/run/cloudera-scm-agent/process/52-hdfs-NAMENODE/dfs_hosts_allow.txt
    2012-12-02 10:32:25,362 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
    dfs.block.invalidate.limit=1000
    2012-12-02 10:32:25,422 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    dfs.block.access.token.enable=false
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    defaultReplication = 1
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    maxReplication
    = 512
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    minReplication
    = 1
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    maxReplicationStreams = 2
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    shouldCheckForEnoughRacks = true
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    replicationRecheckInterval = 3000
    2012-12-02 10:32:25,441 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    encryptDataTransfer = false
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
    =
    john (auth:SIMPLE)
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    supergroup
    =
    john
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    isPermissionEnabled
    =
    true
    2012-12-02 10:32:25,454 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA
    Enabled:
    false
    2012-12-02 10:32:25,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append
    Enabled:
    true
    2012-12-02 10:32:27,741 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: Caching
    file names
    occuring
    more than 10 times
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.threshold-pct = 0.9990000128746033
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.min.datanodes = 0
    2012-12-02 10:32:27,743 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    dfs.namenode.safemode.extension = 30000
    2012-12-02 10:32:27,941 INFO
    org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /home/john/dfs_dir/namenode/data1/in_use.lock
    acquired by
    nodename
    2070@ubuntu1
    2012-12-02 10:32:28,574 INFO
    org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /home/john/dfs_dir/namenode/data2/in_use.lock
    acquired by
    nodename
    2070@ubuntu1
    2012-12-02 10:32:28,803 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Recovering
    unfinalized segments in
    /home/john/dfs_dir/namenode/data1/current
    2012-12-02 10:32:28,803 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Recovering
    unfinalized segments in
    /home/john/dfs_dir/namenode/data2/current
    2012-12-02 10:32:28,805 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    No edit log streams selected.
    2012-12-02 10:32:28,950 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Loading image file
    /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000
    using
    no compression
    2012-12-02 10:32:28,950 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Number of files = 1
    2012-12-02 10:32:28,951 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Number of files under construction = 0
    2012-12-02 10:32:28,952 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Image file of size 117 loaded in 0 seconds.
    2012-12-02 10:32:28,952 INFO
    org.apache.hadoop.hdfs.server.namenode.FSImage:
    Loaded image for txid 0 from
    /home/john/dfs_dir/namenode/data1/current/fsimage_0000000000000000000
    2012-12-02 10:32:28,955 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting
    log
    segment
    at 1
    2012-12-02 10:32:30,041 INFO
    org.apache.hadoop.hdfs.server.namenode.NameCache:
    initialized with 0
    entries
    0 lookups
    2012-12-02 10:32:30,041 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    Finished loading
    FSImage in 2298 msecs
    2012-12-02 10:32:31,360 INFO org.apache.hadoop.ipc.Server:
    Starting
    Socket
    Reader #1 for port 8020
    2012-12-02 10:32:31,402 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    Registered
    FSNamesystemState MBean
    2012-12-02 10:32:31,482 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number
    of blocks
    under
    construction: 0
    2012-12-02 10:32:31,482 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    initializing
    replication queues
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Total
    number
    of
    blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number of
    invalid blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number of
    under-replicated blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number of
    over-replicated blocks = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    Number of
    blocks
    being written = 0
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.StateChange:
    STATE*
    Replication Queue initialization scan for invalid, over- and
    under-replicated blocks completed in 31 msec
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.StateChange:
    STATE*
    Leaving safe mode after 6 secs.
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.StateChange:
    STATE*
    Network topology has 0 racks and 0 datanodes
    2012-12-02 10:32:31,513 INFO
    org.apache.hadoop.hdfs.StateChange:
    STATE*
    UnderReplicatedBlocks has 0 blocks
    2012-12-02 10:32:31,674 INFO org.mortbay.log: Logging to
    org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
    org.mortbay.log.Slf4jLog
    2012-12-02 10:32:31,814 INFO
    org.apache.hadoop.http.HttpServer: Added
    global
    filter 'safety'
    (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    2012-12-02 10:32:31,815 INFO
    org.apache.hadoop.http.HttpServer: Added
    filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context hdfs
    2012-12-02 10:32:31,815 INFO
    org.apache.hadoop.http.HttpServer: Added
    filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context logs
    2012-12-02 10:32:31,815 INFO
    org.apache.hadoop.http.HttpServer: Added
    filter
    static_user_filter
    (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
    to
    context static
    2012-12-02 10:32:31,841 INFO
    org.apache.hadoop.http.HttpServer:
    dfs.webhdfs.enabled = true
    2012-12-02 10:32:31,842 INFO
    org.apache.hadoop.http.HttpServer: Added
    filter
    'SPNEGO' (class=org.apache.hadoop.hdfs.web.AuthFilter)
    2012-12-02 10:32:31,844 INFO
    org.apache.hadoop.http.HttpServer:
    addJerseyResourcePackage:

    packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
    pathSpec=/webhdfs/v1/*
    2012-12-02 10:32:31,862 INFO
    org.apache.hadoop.http.HttpServer: Jetty
    bound
    to port 50070
    2012-12-02 10:32:31,862 INFO org.mortbay.log:
    jetty-6.1.26.cloudera.2
    2012-12-02 10:32:32,261 WARN
    org.apache.hadoop.security.authentication.server.AuthenticationFilter:
    'signature.secret' configuration not set, using a random
    value as
    secret
    2012-12-02 10:32:32,453 INFO org.mortbay.log: Started
    selectchannelconnector@localhost.localdomain:50070
    2012-12-02 10:32:32,453 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server
    up at:
    localhost.localdomain:50070
    2012-12-02 10:32:32,453 INFO org.apache.hadoop.ipc.Server:
    IPC Server
    Responder: starting
    2012-12-02 10:32:32,454 INFO org.apache.hadoop.ipc.Server:
    IPC Server
    listener on 8020: starting
    2012-12-02 10:32:32,603 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode
    up at:
    ubuntu1/127.0.0.1:8020
    2012-12-02 10:32:32,603 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
    Starting
    services
    required for active state
    2012-12-02 10:32:34,359 INFO
    org.apache.hadoop.fs.TrashPolicyDefault:
    The
    configured checkpoint interval is 0 minutes. Using an
    interval of
    1440
    minutes that is used for deletion instead
    2012-12-02 10:32:35,780 INFO
    org.apache.hadoop.hdfs.StateChange:
    BLOCK*
    NameSystem.registerDatanode: node registration from
    DatanodeRegistration(127.0.0.1,
    storageID=DS-377461142-127.0.0.1-50010-1354473155123,
    infoPort=50075,
    ipcPort=50020,
    storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0)
    storage
    DS-377461142-127.0.0.1-50010-1354473155123
    2012-12-02 10:32:35,790 INFO
    org.apache.hadoop.net.NetworkTopology:
    Adding a
    new node: /default/127.0.0.1:50010
    2012-12-02 10:32:35,951 INFO
    org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
    BLOCK*
    processReport: Received first block report from
    127.0.0.1:50010 after
    becoming active. Its block contents are no longer
    considered stale.
    2012-12-02 10:32:35,951 INFO
    org.apache.hadoop.hdfs.StateChange:
    BLOCK*
    processReport: from DatanodeRegistration(127.0.0.1,
    storageID=DS-377461142-127.0.0.1-50010-1354473155123,
    infoPort=50075,
    ipcPort=50020,
    storageInfo=lv=-40;cid=cluster2;nsid=829593698;c=0),
    blocks:
    0, processing time: 1 msecs
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll
    Edit Log
    from
    127.0.0.1
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling
    edit logs.
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending
    log segment
    1
    2012-12-02 10:33:31,480 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
    transactions: 2
    Total time for transactions(ms): 1Number of transactions
    batched in
    Syncs: 0
    Number of syncs: 2 SyncTimes(ms): 166 189
    2012-12-02 10:33:31,595 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
    transactions: 2
    Total time for transactions(ms): 1Number of transactions
    batched in
    Syncs: 0
    Number of syncs: 3 SyncTimes(ms): 215 255
    2012-12-02 10:33:31,603 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Finalizing
    edits
    file

    /home/john/dfs_dir/namenode/data1/current/edits_inprogress_0000000000000000001
    ->

    /home/john/dfs_dir/namenode/data1/current/edits_0000000000000000001-0000000000000000002
    2012-12-02 10:33:31,604 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
    Finalizing
    edits
    file

    /home/john/dfs_dir/namenode/data2/current/edits_inprogress_0000000000000000001
    ->

    /home/john/dfs_dir/namenode/data2/current/edits_0000000000000000001-0000000000000000002
    2012-12-02 10:33:31,604 INFO
    org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting
    log
    segment
    at 3
    2012-12-02 10:33:33,274 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage:
    Opening
    connection
    to

    http://ubuntu1:50090/getimage?getimage=1&txid=2&storageInfo=-40:829593698:0:cluster2
    2012-12-02 10:33:33,622 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage:
    Transfer took
    0.34s
    at 0.00 KB/s
    2012-12-02 10:33:33,622 INFO
    org.apache.hadoop.hdfs.server.namenode.TransferFsImage:
    Downloaded
    file
    fsimage.ckpt_0000000000000000002 size 117 bytes.
    2012-12-02 10:33:33,841 INFO
    org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager:
    Going
    to
    retain 2 images with txid >= 0



    --
    Harsh J



    --
    Thanks,
    Shouvanik


    --
    Harsh J



    --
    Thanks,
    Shouvanik


    --
    Harsh J

    --
    Thanks,
    *Shouvanik*

    --
    Thanks,
    *Shouvanik*

    --
    Thanks,
    *Shouvanik*

    --
    Thanks,
    *Shouvanik*
  • bc Wong at Dec 4, 2012 at 6:44 pm

    On Tue, Dec 4, 2012 at 10:31 AM, Shouvanik Haldar wrote:

    Yes. I can do a 'hadoop fs -touchz /tmp/foo' without any error.
    By HDFS daemons, do you mean logs that are inside /var/log/hadoop folder?

    If not, can you please tell me where to search for them?
    Your HDFS seems quite unhealthy. The logs are under /var/log/hadoop-hdfs.
    You can also go to CM -> HDFS -> NN/DN -> Processes -> Full log file.

    * Do you see any errors in the NN log?
    * Does the NN web ui show any errors?

    Cheers,
    bc

    *In the log file
    "/var/log/hadoop/hadoop-cmf-mapreduce1-JOBTRACKER-ip-10-40-222-77.ec2.internal.log.out",
    I find the below logs which might be important for you to find out the
    cause.*


    2012-12-04 06:43:28,231 WARN org.apache.hadoop.hdfs.DFSClient:
    DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
    java.io.IOException: File /tmp/mapred/system/jobtracker.info could only
    be replicated to 0 nodes, instead of 1

    2012-12-04 06:43:28,231 WARN org.apache.hadoop.hdfs.DFSClient: Could not
    get block locations. Source file "/tmp/mapred/system/jobtracker.info" -
    Aborting...
    2012-12-04 06:43:28,231 WARN org.apache.hadoop.mapred.JobTracker: Writing
    to file hdfs://ip-10-40-222-77.ec2.internal/tmp/mapred/system/
    jobtracker.info failed!


    2012-12-04 06:43:38,257 WARN org.apache.hadoop.hdfs.DFSClient: Error
    Recovery for block null bad datanode[0] nodes == null
    2012-12-04 06:43:38,257 WARN org.apache.hadoop.hdfs.DFSClient: Could not
    get block locations. Source file "/tmp/mapred/system/jobtracker.info" -
    Aborting...
    2012-12-04 06:43:38,257 WARN org.apache.hadoop.mapred.JobTracker: Writing
    to file hdfs://ip-10-40-222-77.ec2.internal/tmp/mapred/system/
    jobtracker.info failed!


    *And below are logs from
    "/var/log/hadoop/hadoop-cmf-hdfs1-DATANODE-ip-10-40-222-77.ec2.internal.log.out"
    *


    2012-12-04 07:01:14,131 WARN
    org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Call
    to ip-10-40-222-77.ec2.internal/10.40.222.77:8020 failed on local
    exception: java.io.EOFException

    Please help!

    Regards,
    Shouvanik



    On Tue, Dec 4, 2012 at 11:25 PM, bc Wong wrote:

    On Tue, Dec 4, 2012 at 9:02 AM, Shouvanik Haldar <
    shouvanik.haldar@gmail.com> wrote:
    Hi,

    I did that and below is the list of client connections

    tcp 0 0 ::ffff:10.40.222.77:2181 ::ffff:
    10.40.222.77:43512 ESTABLISHED 20441/java
    tcp 0 0 ::ffff:10.40.222.77:2181 ::ffff:
    10.40.222.77:43513 ESTABLISHED 20441/java
    tcp 0 0 ::ffff:10.40.222.77:43512 ::ffff:10.40.222.77:2181
    ESTABLISHED 22681/java
    tcp 0 0 ::ffff:10.40.222.77:43513 ::ffff:10.40.222.77:2181
    ESTABLISHED 22707/java
    tcp 0 0 ::ffff:10.40.222.77:51510 ::ffff:10.40.222.77:2181
    TIME_WAIT -

    Is that a problem?
    Looks ok.

    Please help me folks...I am not able to execute simple hbase
    query..since I am not able to connect to HBase...below I paste all logs
    that I think may give some hint why I am not able to achieve success




    2012-12-04 06:34:56,057 ERROR
    org.apache.hadoop.hbase.master.HMasterCommandLine: Master not running

    ERROR org.apache.hadoop.hdfs.DFSClient: Exception closing file
    /hbase/hbase.version : org.apache.hadoop.ipc.RemoteException:
    java.io.IOException: File /hbase/hbase.version could only be replicated to
    0 nodes, instead of 1
    This error is new (for your cluster). Can you go to your HDFS daemons and
    look for errors in the logs? Can you successfully do a simple `hadoop fs
    -touchz /tmp/foo'?

    Cheers,
    bc

    2012-12-04 09:56:33,009 INFO
    org.apache.hadoop.hbase.master.ActiveMasterManager: Master=
    ec2-75-101-165-36.compute-1.amazonaws.com:60000
    2012-12-04 09:56:33,438 WARN org.apache.hadoop.hdfs.DFSClient:
    DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
    java.io.IOException: File /hbase/hbase.version could only be replicated to
    0 nodes, instead of 1


    2012-12-04 09:56:43,450 WARN org.apache.hadoop.hdfs.DFSClient:
    DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
    java.io.IOException: File /hbase/hbase.version could only be replicated to
    0 nodes, instead of 1

    2012-12-04 09:57:43,533 WARN org.apache.hadoop.hdfs.DFSClient: Error
    Recovery for block null bad datanode[0] nodes == null





    12/12/04 11:13:06 ERROR zookeeper.ZKConfig: no valid quorum servers
    found in zoo.cfg

    ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: An error is
    preventing HBase from connecting to ZooKeeper

    2012-12-04 11:19:06,584 WARN org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b66b4d47c0000, likely client has closed socket

    2012-12-04 11:20:14,000 INFO
    org.apache.zookeeper.server.ZooKeeperServer: Expiring session
    0x13b65bbe40c0013, timeout of 40000ms exceeded

    Got user-level KeeperException when processing
    sessionid:0x13b66b4d47c0001 type:create cxid:0x6 zxid:0xfffffffffffffffe
    txntype:unknown reqpath:n/a Error Path:/hbase/master Error:KeeperErrorCode
    = NodeExists for /hbase/master



    ERROR: org.apache.hadoop.hbase.MasterNotRunningException:
    ec2-75-101-165-36.compute-1.amazonaws.com:60000



    On Tue, Dec 4, 2012 at 12:58 AM, bc Wong wrote:

    On Mon, Dec 3, 2012 at 1:40 AM, Shouvanik Haldar <
    shouvanik.haldar@gmail.com> wrote:
    Hi,

    I performed the steps as you told me. Still I am facing the previously
    pasted ZK exception.
    Where can I find the zookeeper logs that cloudera manager writes to?
    In the admin console, I can find the log location as /var/log/zookeeper. Is
    this the only location?

    It's in /var/log/zookeeper. Can you run `netstat -ntp | grep 2181' on
    your ZK node to see whether there are a lot of client connections?

    Cheers,
    bc

    On Mon, Dec 3, 2012 at 1:42 PM, bc Wong wrote:

    On Sun, Dec 2, 2012 at 11:18 PM, Shouvanik Haldar <
    shouvanik.haldar@gmail.com> wrote:
    Hi,

    Cloudera and all the services require some ports. These are enabled
    in AWS security group. Except that I am not aware of any firewall settings.

    Tell me one more thing. Does the /etc/hosts have anything to do?

    Let me paste the contents of this file.

    *127.0.0.1 localhost
    10.40.222.77 ip-10-40-222-77*


    and for the other server

    *127.0.0.1 localhost
    10.83.35.173 ip-10-83-35-173*

    Is this OK?
    Looks fine. Maybe you should heed this warning:

    "ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase
    is able
    to connect to ZooKeeper but the connection closes immediately. This
    could be
    a sign that the server has too many connections (30 is the default).
    Consider inspecting your ZK server logs for that error and then make
    sure
    you are reusing HBaseConfiguration as often as you can. See HTable's
    javadoc
    for more information."

    Can you restart your ZK and HBase services, and them immediately try
    to do a "describe".

    Another help.
    Can you please point to the exact guide to upgrade to CM 4.x?

    https://ccp.cloudera.com/display/ENT4DOC/Upgrading+to+Cloudera+Manager+4.0

    See the subsections on upgrading from/to different versions.

    Cheers,
    bc

    Regards,
    Shouvanik



    On Mon, Dec 3, 2012 at 12:15 PM, bc Wong wrote:
    On Sun, Dec 2, 2012 at 12:34 PM, Harsh J wrote:

    Hi Shouvanik,

    I notice you use CDH3. You should really be using CDH4, as it
    provides
    a great deal of improvements over the CDH3 version that you'll be
    missing out on.

    In any case:

    If there exists a /etc/zookeeper/zoo.cfg on your client machine,
    please rename it to something else, such as zoo.cfg.unwanted, or
    remove it altogether, and retry.

    Alternatively, try to run your HBase shell command(s) with the
    direct
    script (instead of wrapper /usr/bin/hbase) as:

    /usr/lib/hbase/bin/hbase shell

    And see if it lets you connect and work normally this time.

    On Mon, Dec 3, 2012 at 1:50 AM, Shouvanik Haldar
    wrote:
    Yes Harsh,

    [root@ip-10-40-222-77 ~]# hbase shell
    HBase Shell; enter 'help<RETURN>' for list of supported commands.
    Type "exit<RETURN>" to leave the HBase Shell
    Version 0.90.4-cdh3u3, r, Thu Jan 26 10:13:36 PST 2012

    hbase(main):001:0> describe 't1'

    ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException:
    HBase is able
    to connect to ZooKeeper but the connection closes immediately.
    This could be
    a sign that the server has too many connections (30 is the default).
    Consider inspecting your ZK server logs for that error and then make sure
    you are reusing HBaseConfiguration as often as you can. See
    HTable's javadoc
    for more information.

    Here is some help for this command:
    Describe the named table. For example:
    hbase> describe 't1'


    hbase(main):002:0>


    Secondly,

    When I see the zookeeper logs, it says

    [root@ip-10-40-222-77 ~]# cat
    /var/log/zookeeper/zookeeper-cmf-zookeeper1-SERVER-ip-10-40-222-77.ec2.internal.log
    2012-12-01 23:30:35,726 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from:
    /var/run/cloudera-scm-agent/process/1-zookeeper-server/zoo.cfg
    2012-12-01 23:30:35,731 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no
    config or no
    quorum defined in config, running in standalone mode
    2012-12-01 23:30:35,811 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from:
    /var/run/cloudera-scm-agent/process/1-zookeeper-server/zoo.cfg
    2012-12-01 23:30:35,812 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-01 23:30:35,842 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/1-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-01 23:30:35,843 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/1-zookeeper-server
    2012-12-01 23:30:35,851 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-01 23:30:35,851 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-01 23:30:35,851 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-01 23:30:35,891 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-01 23:30:35,907 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.0
    2012-12-01 23:30:35,941 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog:
    Snapshotting: 28
    2012-12-01 23:38:21,260 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from:
    /var/run/cloudera-scm-agent/process/9-zookeeper-server/zoo.cfg
    2012-12-01 23:38:21,265 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no
    config or no
    quorum defined in config, running in standalone mode
    2012-12-01 23:38:21,339 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from:
    /var/run/cloudera-scm-agent/process/9-zookeeper-server/zoo.cfg
    2012-12-01 23:38:21,340 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/9-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-01 23:38:21,357 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-01 23:38:21,358 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/9-zookeeper-server
    2012-12-01 23:38:21,366 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-01 23:38:21,366 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-01 23:38:21,366 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-01 23:38:21,399 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-01 23:38:21,415 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.28
    2012-12-01 23:38:21,430 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog:
    Snapshotting: 28
    2012-12-01 23:43:49,121 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:45741
    2012-12-01 23:43:49,128 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /
    10.83.35.173:45741
    2012-12-01 23:43:49,130 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.29
    2012-12-01 23:43:49,172 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0000 with negotiated timeout 40000 for
    client /10.83.35.173:45741
    2012-12-01 23:43:52,894 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:59824
    2012-12-01 23:43:52,897 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /
    10.40.222.77:59824
    2012-12-01 23:43:52,899 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0001 with negotiated timeout 40000 for
    client /10.40.222.77:59824
    2012-12-01 23:43:52,993 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:59825
    2012-12-01 23:43:52,996 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /
    10.40.222.77:59825
    2012-12-01 23:43:52,998 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0002 with negotiated timeout 40000 for
    client /10.40.222.77:59825
    2012-12-02 02:48:13,910 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from:
    /var/run/cloudera-scm-agent/process/34-zookeeper-server/zoo.cfg
    2012-12-02 02:48:13,914 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no
    config or no
    quorum defined in config, running in standalone mode
    2012-12-02 02:48:14,002 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from:
    /var/run/cloudera-scm-agent/process/34-zookeeper-server/zoo.cfg
    2012-12-02 02:48:14,003 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-02 02:48:14,030 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-02 02:48:14,030 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-02 02:48:14,030 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/34-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-02 02:48:14,031 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/34-zookeeper-server
    2012-12-02 02:48:14,039 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-02 02:48:14,039 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-02 02:48:14,040 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-02 02:48:14,072 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-02 02:48:14,088 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.28
    2012-12-02 02:48:14,109 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog:
    Snapshotting: 2c
    2012-12-02 02:48:15,025 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:39458
    2012-12-02 02:48:15,029 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0002 at /
    10.40.222.77:39458
    2012-12-02 02:48:15,035 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0002 with negotiated timeout 40000 for
    client /10.40.222.77:39458
    2012-12-02 02:48:15,060 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:56157
    2012-12-02 02:48:15,062 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0000 at /
    10.83.35.173:56157
    2012-12-02 02:48:15,063 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0000 with negotiated timeout 40000 for
    client /10.83.35.173:56157
    2012-12-02 02:48:15,095 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:39459
    2012-12-02 02:48:15,095 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b59e867fc0001 at /
    10.40.222.77:39459
    2012-12-02 02:48:15,096 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b59e867fc0001 with negotiated timeout 40000 for
    client /10.40.222.77:39459
    2012-12-02 03:23:48,488 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.2d
    2012-12-02 03:23:48,560 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0001
    2012-12-02 03:23:48,570 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:39459 which
    had sessionid
    0x13b59e867fc0001
    2012-12-02 03:25:46,004 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54443
    2012-12-02 03:25:46,007 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /
    10.40.222.77:54443
    2012-12-02 03:25:46,009 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30000 with negotiated timeout 40000 for
    client /10.40.222.77:54443
    2012-12-02 03:25:46,348 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30000, likely client has closed socket
    2012-12-02 03:25:46,349 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:54443 which
    had sessionid
    0x13b5a963ea30000
    2012-12-02 03:26:15,044 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b59e867fc0000, likely client has closed socket
    2012-12-02 03:26:15,045 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.83.35.173:56157 which
    had sessionid
    0x13b59e867fc0000
    2012-12-02 03:26:17,867 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b59e867fc0002, likely client has closed socket
    2012-12-02 03:26:17,867 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:39458 which
    had sessionid
    0x13b59e867fc0002
    2012-12-02 03:26:28,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30000, timeout of 40000ms exceeded
    2012-12-02 03:26:28,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5a963ea30000
    2012-12-02 03:26:42,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b59e867fc0000, timeout of 40000ms exceeded
    2012-12-02 03:26:42,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0000
    2012-12-02 03:26:47,391 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54461
    2012-12-02 03:26:47,394 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /
    10.40.222.77:54461
    2012-12-02 03:26:47,396 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30001 with negotiated timeout 40000 for
    client /10.40.222.77:54461
    2012-12-02 03:26:47,448 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:54462
    2012-12-02 03:26:47,460 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /
    10.40.222.77:54462
    2012-12-02 03:26:47,463 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30002 with negotiated timeout 40000 for
    client /10.40.222.77:54462
    2012-12-02 03:26:56,000 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b59e867fc0002, timeout of 40000ms exceeded
    2012-12-02 03:26:56,000 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b59e867fc0002
    2012-12-02 03:27:00,606 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:56794
    2012-12-02 03:27:00,614 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /
    10.83.35.173:56794
    2012-12-02 03:27:00,617 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30003 with negotiated timeout 40000 for
    client /10.83.35.173:56794
    2012-12-02 03:36:45,885 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from:
    /var/run/cloudera-scm-agent/process/43-zookeeper-server/zoo.cfg
    2012-12-02 03:36:45,889 WARN
    org.apache.zookeeper.server.quorum.QuorumPeerMain: Either no
    config or no
    quorum defined in config, running in standalone mode
    2012-12-02 03:36:45,962 INFO
    org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading
    configuration
    from:
    /var/run/cloudera-scm-agent/process/43-zookeeper-server/zoo.cfg
    2012-12-02 03:36:45,962 INFO
    org.apache.zookeeper.server.ZooKeeperServerMain: Starting server
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:zookeeper.version=3.3.4-cdh3u3--1, built on
    01/26/2012
    20:08 GMT
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:host.name=ip-10-40-222-77.ec2.internal
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.version=1.6.0_26
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.vendor=Sun Microsystems Inc.
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.home=/usr/java/jdk1.6.0_26/jre
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.class.path=/var/run/cloudera-scm-agent/process/43-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.4-cdh3u3.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:java.library.path=/usr/java/jdk1.6.0_26/jre/lib/amd64/server:/usr/java/jdk1.6.0_26/jre/lib/amd64:/usr/java/jdk1.6.0_26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.io.tmpdir=/tmp
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:java.compiler=<NA>
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.name=Linux
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.arch=amd64
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:os.version=2.6.32-131.17.1.el6.x86_64
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.name=zookeeper
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server environment:user.home=/var/run/zookeeper
    2012-12-02 03:36:45,976 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Server
    environment:user.dir=/var/run/cloudera-scm-agent/process/43-zookeeper-server
    2012-12-02 03:36:45,985 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    tickTime set to 2000
    2012-12-02 03:36:45,985 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    minSessionTimeout set to 4000
    2012-12-02 03:36:45,985 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    maxSessionTimeout set to 40000
    2012-12-02 03:36:46,024 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    binding to port 0.0.0.0/0.0.0.0:2181
    2012-12-02 03:36:46,042 INFO
    org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
    /var/zookeeper/version-2/snapshot.2c
    2012-12-02 03:36:46,063 INFO
    org.apache.zookeeper.server.persistence.FileTxnSnapLog:
    Snapshotting: 36
    2012-12-02 03:36:46,531 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:37153
    2012-12-02 03:36:46,535 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30003 at /
    10.83.35.173:37153
    2012-12-02 03:36:46,539 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30003 with negotiated timeout 40000 for
    client /10.83.35.173:37153
    2012-12-02 03:36:46,883 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:47487
    2012-12-02 03:36:46,884 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30002 at /
    10.40.222.77:47487
    2012-12-02 03:36:46,884 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30002 with negotiated timeout 40000 for
    client /10.40.222.77:47487
    2012-12-02 03:36:47,310 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:47488
    2012-12-02 03:36:47,311 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to renew session 0x13b5a963ea30001 at /
    10.40.222.77:47488
    2012-12-02 03:36:47,312 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5a963ea30001 with negotiated timeout 40000 for
    client /10.40.222.77:47488
    2012-12-02 03:41:46,616 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44776
    2012-12-02 03:41:46,619 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /
    10.40.222.77:44776
    2012-12-02 03:41:46,620 INFO
    org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file:
    log.37
    2012-12-02 03:41:46,674 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0000 with negotiated timeout 40000 for
    client /10.40.222.77:44776
    2012-12-02 03:41:47,494 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5ac2ad7a0000, likely client has closed socket
    2012-12-02 03:41:47,495 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:44776 which
    had sessionid
    0x13b5ac2ad7a0000
    2012-12-02 03:42:16,001 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30003, likely client has closed socket
    2012-12-02 03:42:16,002 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.83.35.173:37153 which
    had sessionid
    0x13b5a963ea30003
    2012-12-02 03:42:17,916 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30002, likely client has closed socket
    2012-12-02 03:42:17,917 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:47487 which
    had sessionid
    0x13b5a963ea30002
    2012-12-02 03:42:19,468 WARN
    org.apache.zookeeper.server.NIOServerCnxn:
    EndOfStreamException: Unable to read additional data from client sessionid
    0x13b5a963ea30001, likely client has closed socket
    2012-12-02 03:42:19,469 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Closed socket connection for client /10.40.222.77:47488 which
    had sessionid
    0x13b5a963ea30001
    2012-12-02 03:42:28,000 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5ac2ad7a0000, timeout of 40000ms exceeded
    2012-12-02 03:42:28,001 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Processed session
    termination for sessionid: 0x13b5ac2ad7a0000
    2012-12-02 03:42:46,214 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.83.35.173:52486
    2012-12-02 03:42:46,217 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /
    10.83.35.173:52486
    2012-12-02 03:42:46,220 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0001 with negotiated timeout 40000 for
    client /10.83.35.173:52486
    2012-12-02 03:42:47,857 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Accepted socket connection from /10.40.222.77:44799
    2012-12-02 03:42:47,864 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Client attempting to establish new session at /
    10.40.222.77:44799
    2012-12-02 03:42:47,867 INFO
    org.apache.zookeeper.server.NIOServerCnxn:
    Established session 0x13b5ac2ad7a0002 with negotiated timeout 40000 for
    client /10.40.222.77:44799

    2012-12-02 03:42:47,991 INFO
    org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
    KeeperException when processing sessionid:0x13b5ac2ad7a0002
    type:create
    cxid:0x6 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
    Path:/hbase/master Error:KeeperErrorCode = NodeExists for
    /hbase/master
    2012-12-02 03:42:48,001 INFO
    org.apache.zookeeper.server.ZooKeeperServer:
    Expiring session 0x13b5a963ea30002, timeout of 40000ms exceeded
    2012-12-