Mongo secondaries stuck at startup state

Question

I have a MongoDB replica set with secondaries that won't make it past the STARTUP state. The database is large but they should have moved to STARTUP2 by now. The correct ports are open as I can connect to the secondaries from the primary and vice-versa.

Solution

The next steps resolved the similar trouble:

1. On the PRIMARY member:

   rs.status()
   {
   "set" : "ShardD",
   "date" : ISODate("2015-08-28T17:01:40.647Z"),
   "myState" : 1,
   "members" : [
           {
                   "_id" : 0,
                   "name" : "host.example.com:27017",
                   "health" : 1,
                   "state" : 1,
                   "stateStr" : "PRIMARY",
                   "uptime" : 1167,
                   "optime" : Timestamp(1440780623, 1),
                   "optimeDate" : ISODate("2015-08-28T16:50:23Z"),
                   "electionTime" : Timestamp(1440780252, 2),
                   "electionDate" : ISODate("2015-08-28T16:44:12Z"),
                   "configVersion" : 3,
                   "self" : true
           },
           {
                   "_id" : 1,
                   "name" : "192.0.2.222:27017",
                   "health" : 1,
                   "state" : 0,
                   "stateStr" : "STARTUP",
                   "uptime" : 584,
                   "optime" : Timestamp(0, 0),
                   "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
                   "lastHeartbeat" : ISODate("2015-08-28T17:01:40.601Z"),
                   "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
                   "pingMs" : 0,
                   "configVersion" : -2
           },
           {
                   "_id" : 2,
                   "name" : "192.0.2.223:27017",
                   "health" : 1,
                   "state" : 0,
                   "stateStr" : "STARTUP",
                   "uptime" : 676,
                   "lastHeartbeat" : ISODate("2015-08-28T17:01:40.643Z"),
                   "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
                   "pingMs" : 0,
                   "configVersion" : -2
           }
   ],
   "ok" : 1
   

   }

2. Check a name of the PRIMARY member (usually it is equal"_id" : 0)

3. If the name has a domain name form, try rewrite the name to IP address. Change Hostnames in a Replica Set:

   cfg = rs.conf()
   cfg.members[0].host = "192.0.2.221"
   rs.reconfig(cfg)
   

After it replica set came to normal status.

P.S. host.example.com was resolved to IP address on all hosts.

Comments

[emmdee]

I realize this is very old question but is this still the case? You cannot use hostnames in your replica config? I'm having the same issue and REALLY don't want to hardcode IP addresses in any config.

[Dimaf]

@emmdee: Mongo >= 3.0.0, at least, doesn't have this bug. FQDN should be set for host in the replica config.