Nodejs/express, shutdown gracefully

Question

I have an nodejs server running express to server out http calls. Is there a recommended approach to shutting down the server gracefully if something bad happens? Should I leave the server running somehow?

I.E. on an uncaught exception the server just stops, I think this will kill over connected clients and not hand them back a response.

Should I:

1. Wait for all http connections to complete before I allow the server to die (and then restart it).
2. Or should I try and stop the server from ever dieing?

is this proper?

process.on('exit', function () {
  console.log('About to exit, waiting for remaining connections to complete');
  app.close();
});

In this case an error might have been thrown leaving the server in an undefined state and the server will continue to finish up remaining connections.

Is there any good way to handle errors keep running or should I let the server die and restart?

Solution 1

Don't try to do anything fancy with unhandled exceptions. Let the server die.

Generally, if a route handler throws an exception, Express will simply catch it and return a HTTP 500 to the client. Since Express caught the exception, your server will not crash.

What usually happens that brings down a server is when an exception gets thrown inside a callback; for example:

app.get('/foo', function(req, res) {
    db.query(..., function(err, r) {
        throw new Error(); // oops, we'll crash
    });
});

Because the callback naturally executes outside the Express router call stack, there's no way to associate it with a specific request. However, you can guard against such a situation:

app.get('/foo', function(req, res, next) {
    db.query(..., function(err, r) {
        try {
            throw new Error();
        } catch(ex) {
            next(ex);
        }
    });
});

Express' next function takes an error as its first argument. If you call next with an error argument, it will stop processing routes and look for an error handler, or just return a 500.

Of course, wrapping everything in try/catch is probably overkill; most things don't actually throw exceptions. You should only do this if you know something might throw. Otherwise, you might end up in a very difficult to debug inconsistent state by swallowing exceptions.

It's important to remember that once an unexpected exception is thrown, your application is in an undefined state. It's possible that the problem request will never complete, and your app would never restart. Or any number of other strange things could happen. (Was it a database issue? Filesystem? Corrupted memory?)

These kinds of situations are incredibly difficult to diagnose, much less debug. It's safer and arguably better to just fail fast, crash, and restart the server quickly. Yes, any clients that happen to be connected would get cut off, but it's easy enough for them to simply retry.

If you're worried about availability, use the cluster module so that you have several (usually number of CPU cores) server processes running, and one crashing won't bring the entire site down.

Solution 2

NodeJS will shutdown when it's not expecting anything to happen; While the http server is waiting on connections node stays running.

server.close([callback])

Stops the server from accepting new connections and keeps existing connections. This function is asynchronous, the server is finally closed when all connections are ended and the server emits a close event. Optionally, you can pass a callback to listen for the close event.

After you do this the server will continue running until the last connection has finished and then shutdown gracefully.

If you don't want to wait for connections to terminate you can use socket.end() or socket.destroy() on your remaining connections.

See http://nodejs.org/api/net.html

---

process.on('exit', function () {
    console.log('About to exit, waiting for remaining connections to complete');
    app.close();
});

This function should run when node exists and not actually tell node to exit. Here is one way you can end your process but it wouldn't end gracefully:

process.exit([code])

Ends the process with the specified code. If omitted, exit uses the 'success' code 0.

To exit with a 'failure' code:

process.exit(1); The shell that executed node should see the exit code as 1.

See http://nodejs.org/api/process.html

---

I hope this helps!

Comments

[lostintranslation]

This is a great explanation. One thing I am still unclear about though. 'It's important to remember that once an unexpected exception is thrown, your application is in an undefined state. It's possible that the problem request will never complete, and your app would never restart'. That makes it sound as if I should not call app.close(); on process 'exit'. True?

[josh3736]

Calling app.close() in the process exit handler is pointless: the docs say "The main event loop will no longer be run after the 'exit' callback finishes." In other words, the exit event is telling you the process will end after your function returns. Closing a network listener is pointless since there is no next tick; the code to accept an incoming connection will never have the opportunity to run.

[BorisOkunskiy]

I'm pretty sure you have to listen for SIGTERM and call server.close() there. Read this tutorial for details.

[josh3736]

@incarnate: You won't get a SIGTERM if your server is going down due to an unhandled exception.

[Richard Marr]

This answer is spot on but starting to date a bit. I'd now recommend using Domains to catch unhandled errors so that you can tie then back to the request even when they occur in another call stack. Then the choice is between whether to let the server die immediately, or to shut down gracefully (i.e. let existing connections complete), and for me that would depend on whether you're storing any state, or have sensitive interactions that don't get validated at the other end.

[poshest]

@RichardMarr "so that you can tie them back to the request even when they occur in another call stack": doesn't passing the error to the callback achieve this in the Express context? Are you saying Domains should be used because there might be other uncaught errors that josh's method won't catch, or that Domains alleviate the need to put try...catch everywhere?

[Richard Marr]

@poshest there might be unhandled errors in external dependencies that you just can't capture with a try/catch. With sensible use of domains you'd just need try/catch in areas with known risks, or where you want unhandled errors to have specific behaviour, e.g. using JSON.parse() on an external API response.

[doug65536]

I like the advice not to swallow exceptions arbitrarily. Exceptions are your best friends when debugging.

[geisterfurz007]

Please add a node showing affiliation :) According to npm you are maintainer of the linked plugin.