How to handle Mongoose DB connection interruptions

Question

I've been evaluating Mongoose (an ORM for node.js which uses MongoDB for persistent storage).

What I'd like to do is make sure that the app can run when the DB is not up when the app starts, and also handles the DB going down intelligently.

Currently my test app which does not work in either case does this:

var mongoose_connection = mongoose.createConnection(DATABASE_URL, {server:{poolSize:4}});

Then I use that connection when making Models.

In the case when the DB is down at the start of the app then any save() calls on instances silently fail with no error. If the DB comes back up they never get written.

So I would need to detect that the connection never happened and have the app be able to tell that at runtime so that I can handle it somehow.

When the DB goes down after the app has started though the save() calls still do not cause errors but they are queued and the written when the DB comes back.

That seems fine except I'd like to hook into the API to get events when the DB is down and to query how many save's are queued. At some point I might have so many queued events that I would want to just stop making new ones and have the app back off.

Solution 1

Case #1: db is down on app startup. there is a minor bug preventing this use case that I'm fixing now. However, here is the work-around:

var db = mongoose.createConnection();
db.on('error', function (err) {
  if (err) // couldn't connect

  // hack the driver to allow re-opening after initial network error
  db.db.close();

  // retry if desired
  connect();
});

function connect () {
  db.open('localhost', 'dbname');
}

connect();

https://gist.github.com/2878607

An ugly but working gist. First shutdown mongo, then run this gist.

Notice the connection failures.

Then start mongo and see all of the queued up inserts complete and dumped to the console. Writes and finds will begin.

Shutdown mongo, notice that the inserts and finds are being attempted but no callbacks are executed.

Restart mongo. Notice all of the queued inserts and finds are completed.

Solution 2

If you would rather fail all the requests to the server when the db is down the native driver does emit the reconnect event which can be sensed in a middleware.

This works and emits the reconnect event fine (mongodb native driver 1.3.23)

mongoose.connection.db.on('reconnect', function (ref) {
    connected=true;
    console.log('reconnect to mongo server.');
});

So my dbconnection middleware looks for connected/error/reconnect (some of the events are redundant but does not harm !) PS. the initial connect failure needs to still be handled by a retry as aaronheckmann's answer above.

mongoose.connection.on('open', function (ref) {
    connected=true;
    console.log('open connection to mongo server.');
});

mongoose.connection.on('connected', function (ref) {
    connected=true;
    console.log('connected to mongo server.');
});

mongoose.connection.on('disconnected', function (ref) {
    connected=false;
    console.log('disconnected from mongo server.');
});

mongoose.connection.on('close', function (ref) {
    connected=false;
    console.log('close connection to mongo server');
});

mongoose.connection.on('error', function (err) {
    connected=false;
    console.log('error connection to mongo server!');
    console.log(err);
});

mongoose.connection.db.on('reconnect', function (ref) {
    connected=true;
    console.log('reconnect to mongo server.');
});

Comments

[justinhj]

That works for me and seems to be what I needed to work from, thanks!

[ldonoso]

I'd say setting connected=false on error might be not correct. It seems there are situations where you get an error without being disconnected: > error: Emitted if an error occurs on a connection, like a parseError due to malformed data or a payload larger than 16MB.