nginx critical error with SSL handshaking

Question

I have problem with my nginx on Ubuntu 14.04 LTS. From time to time I get a critical error:

2015/01/18 12:59:44 [crit] 1065#0: *28289 SSL_do_handshake() failed (SSL: error:140A1175:SSL routines:SSL_BYTES_TO_CIPHER_LIST:inappropriate fallback) while SSL handshaking, client: 10.0.2.2, server: 0.0.0.0:443

I've checked version of my OpenSSL:

root@www:~# ldd `which nginx` | grep ssl
        libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f39e236b000)

root@www:~# strings /lib/x86_64-linux-gnu/libssl.so.1.0.0 | grep "^OpenSSL "
OpenSSL 1.0.1f 6 Jan 2014

I've searched for more information about it and found that it might be problem with old version OpenSSL. So I've tried to compile the latest version:

wget https://www.openssl.org/source/openssl-1.0.1l.tar.gz && tar xzf && cd openssl-1.0.1l

./config && make && make install

I've also replaced old OpenSSL binary file with new one via symlink:

ln -sf /usr/local/ssl/bin/openssl `which openssl`

After that I have:

root@www:~# openssl version
OpenSSL 1.0.1l 15 Jan 2015

But still I have the old version in nginx:

root@www:~# strings /lib/x86_64-linux-gnu/libssl.so.1.0.0 | grep "^OpenSSL "
OpenSSL 1.0.1f 6 Jan 2014

I couldn't find any other new libssl in Ubuntu after updating OpenSSL. How do I update libssl so that nginx could use the newest version?

P.S.1. Maybe the problem with critical error isn't about version of OpenSSL.

P.S.2. I think that this crtitical error might affect my whole Virtual Machine. I have also a problem with "from time to time" crashing of VM.

I've tried so many things and now I am hopeless. Stackoverflow please help!

Solution

...BYTES_TO_CIPHER_LIST:inappropriate fallback) while SSL handshaking, client: 10.0.2.2, server: 0.0.0.0:443

This looks like someone checking if the server supports TLS_FALLBACK_SCSV, which it does in your case. Nothing to worry about. On the contrary this means that your server supports a useful security feature. For more information about TLS_FALLBACK_SCSV and how one can detect SSL downgrade attacks like POODLE this way you might have a look at http://www.exploresecurity.com/poodle-and-the-tls_fallback_scsv-remedy/.

TLS_FALLBACK_SCSV is a fairly new option intended to detect SSL downgrade attacks. It needs support on client and server. Older nginx/OpenSSL and older browsers simply did not have this option so this problem could not have been detected and thus not logged in earlier versions. This message is critical because it could indicate an actual SSL downgrade attack attempt against the client which was defeated by this option. In practice it is probably some tool probing for support of the option, like SSLLabs.

For reference the relevant code from ssl/ssl_lib.c function ssl_bytes_to_cipher_list:

/* Check for TLS_FALLBACK_SCSV */
if ((n != 3 || !p[0]) &&
        (p[n-2] == ((SSL3_CK_FALLBACK_SCSV >> 8) & 0xff)) &&
        (p[n-1] == (SSL3_CK_FALLBACK_SCSV & 0xff)))
        {
        /* The SCSV indicates that the client previously tried a higher version.
         * Fail if the current version is an unexpected downgrade. */
        if (!SSL_ctrl(s, SSL_CTRL_CHECK_PROTO_VERSION, 0, NULL))
                {
                SSLerr(SSL_F_SSL_BYTES_TO_CIPHER_LIST,SSL_R_INAPPROPRIATE_FALLBACK);
                if (s->s3)
                        ssl3_send_alert(s,SSL3_AL_FATAL,SSL_AD_INAPPROPRIATE_FALLBACK);
                goto err;
                }
        p += n;
        continue;
        }

Comments

[MegaKaskaskas]

I've never had this problem before. Earlier I was using Debian with older nginx. If this problem is tagged as critical in nginx log then what's critical about it? Your answer is clear that there is no problem at all. I don't get it.

[Steffen Ullrich]

I've updated the answer to include more information.

[MegaKaskaskas]

Thank you for explaining the problem @Steffen Ullrich. After manually updating OpenSSL packages, and nginx on Ubuntu the problem still occurs. During daily activity nginx generates dozens of error logs with the following contents: 2015/01/27 10:06:15 p.m. [crit] 730 # 0: * 263,168 SSL_do_handshake () failed (SSL: error: 140A1175: SSL routines: SSL_BYTES_TO_CIPHER_LIST: Inappropriate fallback) while SSL handshaking, client: 10.0.2.2, server: 0.0.0.0:443

[MegaKaskaskas]

It does not affect the proper operation of the server on the daily basis, but from time to time the error occurs continuously (string). It jams the error log every other second and eventually blocks all services on the virtual machine (including nginx, php5-fpm, proftpd, nodejs). All I can in this situation is rebooting the system. I realize that what you described should not affect either the functioning of nginx or the operation of the server, but in my case it is completely the other way round. It causes a huge problem that is blocking the website and preventing the users from entering it.

[MegaKaskaskas]

Is there an option to turn off the SSL function which causes these fatal errors?

[Steffen Ullrich]

I don't think there is an option. And are you even sure that this option is causing the problems? I would rather assume, that if this option happens somebody is doing a scan against your site to look for problems. And the load of this scan or other parts of this scan might cause the problems in reality.

[MegaKaskaskas]

I've updated nginx to mainline version 1.7.9. Problem with critical error is resolved, however my VM is still not working properly. Right now I have no errors anywhere and just as I wrote before from time to time all services on virtual machine stops responding to any connections.

[Steffen Ullrich]

I think this is not a result of TLS_FALLBACK_SCSV, but might even the cause for these log messages, because clients failed to connect and thus re-tried with a downgraded TLS version. See also serverfault.com/a/663309/208324.