FAQ
I have a three member mongo replication set. Authentication is enabled
through keyfile. The problem is the slave gets stuck in STARTUP2 state and
does not move further. Seems like the sync starts and then it complains
about not finding a member to sync from. What could be the problem here?
Also am not able to connect to the mongo shell on the slave, its stuck as
well. Any pointers??

Logs from primary

2016-06-08T01:04:38.044+0000 I CONTROL [initandlisten] allocator: tcmalloc2016-06-08T01:04:38.044+0000 I CONTROL [initandlisten] modules: none2016-06-08T01:04:38.044+0000 I CONTROL [initandlisten] build environment:2016-06-08T01:04:38.045+0000 I CONTROL [initandlisten] distmod: rhel702016-06-08T01:04:38.045+0000 I CONTROL [initandlisten] distarch: x86_642016-06-08T01:04:38.045+0000 I CONTROL [initandlisten] target_arch: x86_642016-06-08T01:04:38.045+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { port: 27017 }, operationProfiling: { mode: "all" }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, replication: { replSetName: "some_name" }, security: { keyFile: "/etc/mongodb-keyfile" }, storage: { dbPath: "/var/lib/mongo", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }2016-06-08T01:04:38.066+0000 I - [initandlisten] Detected data files in /var/lib/mongo created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.2016-06-08T01:04:38.066+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),2016-06-08T01:04:38.613+0000 I STORAGE [initandlisten] Starting WiredTigerRecordStoreThread local.oplog.rs2016-06-08T01:04:38.613+0000 I STORAGE [initandlisten] The size storer reports that the oplog contains 7 records totaling to 1275 bytes2016-06-08T01:04:38.613+0000 I STORAGE [initandlisten] Scanning the oplog to determine where to place markers for truncation2016-06-08T01:04:38.667+0000 I CONTROL [initandlisten]2016-06-08T01:04:38.667+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.2016-06-08T01:04:38.667+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'2016-06-08T01:04:38.667+0000 I CONTROL [initandlisten]2016-06-08T01:04:38.667+0000 I CONTROL [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 4096 processes, 64000 files. Number of processes should be at least 32000 : 0.5 times number of files.2016-06-08T01:04:38.667+0000 I CONTROL [initandlisten]2016-06-08T01:04:38.670+0000 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongo/diagnostic.data'2016-06-08T01:04:38.670+0000 I NETWORK [initandlisten] waiting for connections on port 270172016-06-08T01:04:38.686+0000 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker2016-06-08T01:04:38.689+0000 I REPL [ReplicationExecutor] New replica set config in use: { _id: "some_name", version: 1, protocolVersion: 1, members: [ { _id: 0, host: "test-mongo1:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } }2016-06-08T01:04:38.689+0000 I REPL [ReplicationExecutor] This node is test-mongo1:27017 in the config2016-06-08T01:04:38.689+0000 I REPL [ReplicationExecutor] transition to STARTUP22016-06-08T01:04:38.690+0000 I REPL [ReplicationExecutor] transition to SECONDARY2016-06-08T01:04:38.690+0000 I REPL [ReplicationExecutor] conducting a dry run election to see if we could be elected2016-06-08T01:04:38.690+0000 I REPL [ReplicationExecutor] dry election run succeeded, running for election2016-06-08T01:04:38.690+0000 I REPL [ReplicationExecutor] election succeeded, assuming primary role in term 22016-06-08T01:04:38.690+0000 I REPL [ReplicationExecutor] transition to PRIMARY2016-06-08T01:04:38.690+0000 I REPL [rsSync] transition to primary complete; database writes are now permitted2016-06-08T01:04:50.480+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:45877 #1 (1 connection now open)2016-06-08T01:04:50.496+0000 I ACCESS [conn1] Successfully authenticated as principal admin on admin2016-06-08T01:04:50.497+0000 I REPL [conn1] replSetReconfig admin command received from client2016-06-08T01:04:50.522+0000 I REPL [conn1] replSetReconfig config object with 2 members parses ok2016-06-08T01:04:50.545+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to 10.3.0.14:270172016-06-08T01:04:50.546+0000 I REPL [ReplicationExecutor] New replica set config in use: { _id: "some_name", version: 2, protocolVersion: 1, members: [ { _id: 0, host: "test-mongo1:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "10.3.0.14:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } }2016-06-08T01:04:50.546+0000 I REPL [ReplicationExecutor] This node is test-mongo1:27017 in the config


Logs from slave

2016-06-08T01:04:38.661+0000 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongo/diagnostic.data'2016-06-08T01:04:38.661+0000 I NETWORK [initandlisten] waiting for connections on port 270172016-06-08T01:04:38.661+0000 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker2016-06-08T01:04:50.503+0000 I NETWORK [initandlisten] connection accepted from 10.3.0.13:32897 #1 (1 connection now open)2016-06-08T01:04:50.520+0000 I ACCESS [conn1] Successfully authenticated as principal __system on local2016-06-08T01:04:50.523+0000 I NETWORK [conn1] end connection 10.3.0.13:32897 (0 connections now open)2016-06-08T01:04:50.524+0000 I NETWORK [initandlisten] connection accepted from 10.3.0.13:32898 #2 (1 connection now open)2016-06-08T01:04:50.545+0000 I ACCESS [conn2] Successfully authenticated as principal __system on local2016-06-08T01:04:50.584+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to test-mongo1:270172016-06-08T01:04:50.713+0000 I REPL [replExecDBWorker-0] Starting replication applier threads2016-06-08T01:04:50.713+0000 I REPL [ReplicationExecutor] New replica set config in use: { _id: "some_name", version: 2, protocolVersion: 1, members: [ { _id: 0, host: "test-mongo1:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "10.3.0.14:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } }2016-06-08T01:04:50.713+0000 I REPL [ReplicationExecutor] This node is 10.3.0.14:27017 in the config2016-06-08T01:04:50.713+0000 I REPL [ReplicationExecutor] transition to STARTUP22016-06-08T01:04:50.714+0000 I REPL [rsSync] ******2016-06-08T01:04:50.714+0000 I REPL [rsSync] creating replication oplog of size: 1336MB...2016-06-08T01:04:50.714+0000 I REPL [ReplicationExecutor] Member test-mongo1:27017 is now in state PRIMARY2016-06-08T01:04:50.771+0000 I STORAGE [rsSync] Starting WiredTigerRecordStoreThread local.oplog.rs2016-06-08T01:04:50.771+0000 I STORAGE [rsSync] The size storer reports that the oplog contains 0 records totaling to 0 bytes2016-06-08T01:04:50.771+0000 I STORAGE [rsSync] Scanning the oplog to determine where to place markers for truncation2016-06-08T01:04:51.286+0000 I REPL [rsSync] ******2016-06-08T01:04:51.286+0000 I REPL [rsSync] initial sync pending2016-06-08T01:04:51.398+0000 I REPL [ReplicationExecutor] syncing from: test-mongo1:270172016-06-08T01:04:51.974+0000 I REPL [rsSync] initial sync drop all databases2016-06-08T01:04:51.974+0000 I STORAGE [rsSync] dropAllDatabasesExceptLocal 12016-06-08T01:04:51.974+0000 I REPL [rsSync] initial sync clone all databases2016-06-08T01:04:51.975+0000 I REPL [rsSync] initial sync cloning db: admin2016-06-08T01:04:52.224+0000 I INDEX [rsSync] build index on: admin.system.users properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "admin.system.users" }2016-06-08T01:04:52.224+0000 I INDEX [rsSync] building index using bulk method2016-06-08T01:04:52.260+0000 I INDEX [rsSync] build index done. scanned 2 total records. 0 secs2016-06-08T01:04:52.380+0000 I INDEX [rsSync] build index on: admin.system.version properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "admin.system.version" }2016-06-08T01:04:52.380+0000 I INDEX [rsSync] building index using bulk method2016-06-08T01:04:52.402+0000 I INDEX [rsSync] build index done. scanned 1 total records. 0 secs2016-06-08T01:04:52.403+0000 I REPL [rsSync] initial sync data copy, starting syncup2016-06-08T01:04:52.403+0000 I REPL [rsSync] oplog sync 1 of 32016-06-08T01:04:52.404+0000 I REPL [rsSync] oplog sync 2 of 32016-06-08T01:04:52.405+0000 I REPL [rsSync] initial sync building indexes2016-06-08T01:04:52.405+0000 I REPL [rsSync] initial sync cloning indexes for : admin2016-06-08T01:04:52.427+0000 I STORAGE [rsSync] copying indexes for: { name: "system.users", options: {} }2016-06-08T01:04:52.521+0000 I INDEX [rsSync] build index on: admin.system.users properties: { v: 1, unique: true, key: { user: 1, db: 1 }, name: "user_1_db_1", ns: "admin.system.users" }2016-06-08T01:04:52.521+0000 I INDEX [rsSync] building index using bulk method2016-06-08T01:04:52.543+0000 I INDEX [rsSync] build index done. scanned 2 total records. 0 secs2016-06-08T01:04:52.543+0000 I STORAGE [rsSync] copying indexes for: { name: "system.version", options: {} }2016-06-08T01:04:52.544+0000 I REPL [rsSync] oplog sync 3 of 32016-06-08T01:04:52.545+0000 I - [rsSync] Creating profile collection: admin.system.profile2016-06-08T01:04:52.715+0000 I REPL [ReplicationExecutor] could not find member to sync from


Configuration

# mongod.conf
# where to write logging data.
systemLog:
   destination: file
   logAppend: true
   path: /var/log/mongodb/mongod.log
# Where and how to store data.
storage:
   dbPath: /var/lib/mongo
   journal:
     enabled: true# engine:# mmapv1:# wiredTiger:
# how the process runs
processManagement:
   fork: true # fork and run in background
   pidFilePath: /var/run/mongodb/mongod.pid # location of pidfile
# network interfaces
net:
   port: 27017
  #bindIp: 127.0.0.1 # Listen to local interface only, comment to listen on all interfaces.


security:
   keyFile: /etc/mongodb-keyfile

operationProfiling:
   mode: all

replication:
   replSetName: somename
#sharding:
## Enterprise-Only Options
#auditLog:
#snmp:


--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.org/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@googlegroups.com.
To post to this group, send email to mongodb-user@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/07f1d59c-d846-4683-ae9f-c1362d7cad11%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 1 of 1 | next ›
Discussion Overview
groupmongodb-user @
categoriesmongodb
postedJun 15, '16 at 5:33a
activeJun 15, '16 at 5:33a
posts1
users1
websitemongodb.org
irc#mongodb

1 user in discussion

Arvind T: 1 post

People

Translate

site design / logo © 2022 Grokbase