MongoConnectionException - No candidate servers found

Question

I am developing a PHP web application using a MongoDB replicaset to store my data. I occasionally receive the following error:

Fatal error: Uncaught exception 'MongoConnectionException' with message 'No candidate servers found'

I have a 3 member Mongo replica set with 1 arbiter

rs0:PRIMARY> rs.status()
{
    "set" : "rs0",
    "date" : ISODate("2013-01-30T01:04:04Z"),
    "myState" : 1,
    "members" : [
        {
            "_id" : 0,
            "name" : "JenEricsMacPro.local:27017",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 844478,
            "optime" : Timestamp(1359507378000, 1),
            "optimeDate" : ISODate("2013-01-30T00:56:18Z"),
            "self" : true
        },
        {
            "_id" : 1,
            "name" : "ericsmacbookpro.local:27017",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 10720,
            "optime" : Timestamp(1359507378000, 1),
            "optimeDate" : ISODate("2013-01-30T00:56:18Z"),
            "lastHeartbeat" : ISODate("2013-01-30T01:04:04Z"),
            "pingMs" : 3
        },
        {
            "_id" : 2,
            "name" : "ericsmacbookair.local:27017",
            "health" : 1,
            "state" : 7,
            "stateStr" : "ARBITER",
            "uptime" : 1206,
            "lastHeartbeat" : ISODate("2013-01-30T01:04:03Z"),
            "pingMs" : 4
        },
        {
            "_id" : 3,
            "name" : "ericsmacxps.local:27017",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 75204,
            "optime" : Timestamp(1359507378000, 1),
            "optimeDate" : ISODate("2013-01-30T00:56:18Z"),
            "lastHeartbeat" : ISODate("2013-01-30T01:04:03Z"),
            "pingMs" : 1
        }
    ],
    "ok" : 1
}

Below is an example of the code where I occasionally receive this error.

$mongoConfig = array (
    'uri' => 'mongodb://JenEricsMacPro.local:27017,ericsmacbookpro.local:27017,ericsmacxps.local:27017/',
    'database' => 'myDatabase',
    'replicaSetArray' => array('replicaSet' => 'rs0'));

$connection = new MongoClient(
    $mongoConfig['uri'],
    $mongoConfig['replicaSetArray']);

$connection->setReadPreference(MongoClient::RP_SECONDARY_PREFERRED);

$db = $connection->selectDB($mongoConfig['database']);

Anyone have any idea what might be causing the 'No candidate servers found' message or any suggestions on how I could determine the root of the problem?

All of my mongo.conf files are using the static IP of each server, no firewalls and DNS appears to be resolving correctly to the .local domain on my network.

EDIT: I added logging, suggested by Derick, prior to instantiating MongoClient.

error_reporting (E_NOTICE);

MongoLog::setModule( MongoLog::ALL );
MongoLog::setLevel( MongoLog::ALL );

After enabling logging I received the following notices.

Notice: PARSE INFO: Parsing mongodb://JenEricsMacPro.local:27017,ericsmacbookpro.local:27017,ericsmacxps.local:27017/ in MongoLogic.php on line 27
Notice: PARSE INFO: - Found node: JenEricsMacPro.local:27017 in MongoLogic.php on line 27
Notice: PARSE INFO: - Found node: ericsmacbookpro.local:27017 in MongoLogic.php on line 27
Notice: PARSE INFO: - Found node: ericsmacxps.local:27017 in MongoLogic.php on line 27
Notice: PARSE INFO: - Connection type: MULTIPLE in MongoLogic.php on line 27
Notice: PARSE INFO: - Found option 'replicaSet': 'rs0' in MongoLogic.php on line 27
Notice: PARSE INFO: - Switching connection type: REPLSET in MongoLogic.php on line 27
Notice: CON INFO: mongo_get_read_write_connection: finding a REPLSET connection (read) in MongoLogic.php on line 27
Notice: CON FINE: found connection JenEricsMacPro.local:27017;rs0;X;297 (looking for JenEricsMacPro.local:27017;rs0;X;297) in MongoLogic.php on line 27
Notice: CON INFO: is_ping: pinging JenEricsMacPro.local:27017;rs0;X;297 in MongoLogic.php on line 27
Notice: CON FINE: mongo_connection_destroy: Closing socket for JenEricsMacPro.local:27017;rs0;X;297. in MongoLogic.php on line 27
Notice: CON INFO: freeing connection JenEricsMacPro.local:27017;rs0;X;297 in MongoLogic.php on line 27
Notice: CON WARN: Couldn't connect to 'JenEricsMacPro.local:27017': send_package: error reading from socket: Operation timed out in MongoLogic.php on line 27
Notice: CON FINE: found connection ericsmacbookpro.local:27017;rs0;X;297 (looking for ericsmacbookpro.local:27017;rs0;X;297) in MongoLogic.php on line 27
Notice: CON INFO: is_ping: pinging ericsmacbookpro.local:27017;rs0;X;297 in MongoLogic.php on line 27
Notice: CON FINE: mongo_connection_destroy: Closing socket for ericsmacbookpro.local:27017;rs0;X;297. in MongoLogic.php on line 27
Notice: CON INFO: freeing connection ericsmacbookpro.local:27017;rs0;X;297 in MongoLogic.php on line 27
Notice: CON WARN: Couldn't connect to 'ericsmacbookpro.local:27017': send_package: error reading from socket: Operation timed out in MongoLogic.php on line 27
Notice: CON FINE: found connection ericsmacxps.local:27017;rs0;X;297 (looking for ericsmacxps.local:27017;rs0;X;297) in MongoLogic.php on line 27
Notice: CON INFO: is_ping: pinging ericsmacxps.local:27017;rs0;X;297 in MongoLogic.php on line 27
Notice: CON FINE: mongo_connection_destroy: Closing socket for ericsmacxps.local:27017;rs0;X;297. in MongoLogic.php on line 27
Notice: CON INFO: freeing connection ericsmacxps.local:27017;rs0;X;297 in MongoLogic.php on line 27
Notice: CON WARN: Couldn't connect to 'ericsmacxps.local:27017': send_package: error reading from socket: Operation timed out in MongoLogic.php on line 27
Notice: CON FINE: discover_topology: checking ismaster for JenEricsMacPro.local:27017;rs0;X;297 in MongoLogic.php on line 27
Notice: CON WARN: discover_topology: couldn't create a connection for JenEricsMacPro.local:27017;rs0;X;297 in MongoLogic.php on line 27
Notice: CON FINE: discover_topology: checking ismaster for ericsmacbookpro.local:27017;rs0;X;297 in MongoLogic.php on line 27
Notice: CON WARN: discover_topology: couldn't create a connection for ericsmacbookpro.local:27017;rs0;X;297 in MongoLogic.php on line 27
Notice: CON FINE: discover_topology: checking ismaster for ericsmacxps.local:27017;rs0;X;297 in MongoLogic.php on line 27
Notice: CON WARN: discover_topology: couldn't create a connection for ericsmacxps.local:27017;rs0;X;297 in MongoLogic.php on line 27
Notice: REPLSET FINE: finding candidate servers in MongoLogic.php on line 27
Notice: REPLSET FINE: - all servers in MongoLogic.php on line 27
Notice: REPLSET FINE: filter_connections: adding connections: in MongoLogic.php on line 27
Notice: REPLSET FINE: filter_connections: done in MongoLogic.php on line 27
Notice: REPLSET FINE: limiting to servers with same replicaset name in MongoLogic.php on line 27
Notice: REPLSET FINE: limiting to servers with same replicaset name: done in MongoLogic.php on line 27
Notice: REPLSET FINE: limiting by credentials in MongoLogic.php on line 27
Notice: REPLSET FINE: limiting by credentials: done in MongoLogic.php on line 27

Solution

I would suggest you add logging to find out what is happening. The Mongo driver (since version 1.3) provides extensive logging to debug connection issues. There is an article at http://derickrethans.nl/mongodb-debugging.html to explain how to turn it on. If from the log info it is not clear what happens, update your question with the log, so that I can update my answer.

From the posted log I can see that your client can not connect to any of the seeds from your connection string. If your machine/client can't connect, you can't talk to MongoDB of course. In this case, there can be a few possibilities:

• The driver can't convert the names to IP addresses. In that case, you need to fix your DNS setup, or add all hosts to /etc/hosts
• You really can't connect to them, this could be a firewall in the way.
• There are some issues with that in the 1.2 series of the driver, so if you're not using 1.3.4 or higher yet, I'd suggest you upgrade. It should be a much smoother experience.
• Remember that you need a majority of your nodes up in order to be able to write. If you don't have that, then you will get the "No candidate servers" error.