Node.js - server closed the connection?

Question

I'm running a web app on a Node.js server and I need it to be online all the time, so I'm using forever. But here is what I'm getting after a period of time:

Error: Connection lost: The server closed the connection.
    at Protocol.end (/home/me/private/app/node_modules/mysql/lib/protocol/Protocol.js:73:13)
    at Socket.onend (stream.js:79:10)
    at Socket.EventEmitter.emit (events.js:117:20)
    at _stream_readable.js:910:16
    at process._tickCallback (node.js:415:13)
error: Forever detected script exited with code: 8
error: Forever restarting script for 3 time

I have two more servers that have been running for about 10 days straight. I have a "keepalive" loop on all servers, making a "select 1" mysql query every 5 minutes or so, but it seems like it doesn't make any difference.

Any ideas?

EDIT 1

My other servers were giving a similar error, I think it was "connection timeout", so I put this function:

function keepalive() {
    db.query('select 1', [], function(err, result) {
        if(err) return console.log(err);    
        console.log('Successful keepalive.');
    });
}

And it fixed my other two servers. But on my main server I'm still getting the error above.

Here is how I'm starting my main server:

var https = require('https');
https.createServer(options, onRequest).listen(8000, 'mydomain.com');

I'm not sure what code are you interested in seeing. Basically the server is a REST API and it needs to stay up all the time. It's getting about 2-5, maybe 10 requests per minute.

Solution

The error isn't related to your HTTPS instance, it's related to your MySQL connection.

The connection to the database is ending unexpectedly and isn't being handled. To fix this, you can use a manual reconnect solution, or you can use connection pooling which automatically handles reconnects.

Here is the manual reconnect example taken from node-mysql's documentation.

var db_config = {
  host: 'localhost',
    user: 'root',
    password: '',
    database: 'example'
};

var connection;

function handleDisconnect() {
  connection = mysql.createConnection(db_config); // Recreate the connection, since
                                                  // the old one cannot be reused.

  connection.connect(function(err) {              // The server is either down
    if(err) {                                     // or restarting (takes a while sometimes).
      console.log('error when connecting to db:', err);
      setTimeout(handleDisconnect, 2000); // We introduce a delay before attempting to reconnect,
    }                                     // to avoid a hot loop, and to allow our node script to
  });                                     // process asynchronous requests in the meantime.
                                          // If you're also serving http, display a 503 error.
  connection.on('error', function(err) {
    console.log('db error', err);
    if(err.code === 'PROTOCOL_CONNECTION_LOST') { // Connection to the MySQL server is usually
      handleDisconnect();                         // lost due to either server restart, or a
    } else {                                      // connnection idle timeout (the wait_timeout
      throw err;                                  // server variable configures this)
    }
  });
}

handleDisconnect();

Comments

[Nikolay Dyankov]

Sorry for the late reply. I just replaced my code with your answer and I'm starting the server. We should know by tomorrow if it worked :) Cheers

[JAR.JAR.beans]

Did it work? also, @hexacyanide , any thoughts on why this would even be needed? why is the MySql connection keeps getting lost sporadically?

[Chris Moore]

I just came across this error for the first time. After months of uptime the MySQL connection was suddenly unavailable. It turned out I had a package called 'unattended-upgrades' installed, and it arranged for a MySQL security update to be applied automatically, which resulted in a MySQL server restart.