Nginx server closes ssl connection for some clients

Question

I have been having some problems with https connections on a box running nginx. Most browsers I have tried work, but some Safari-versions, and openssl fails to connect. https://www.ssllabs.com/ssltest/analyze.html?d=test.relis.no show many attemts as "Server closed connection"

The problem seems to be connected to cipher suits based on some things I have read while googling log content.

When connecting with openssl s_client -connect test.relis.no:443: CONNECTED(00000003)

write:errno=104

no peer certificate available

No client certificate CA names sent

SSL handshake has read 0 bytes and written 295 bytes

New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE

Expansion: NONE

Nginx config:

server {
        listen       443 ssl;
        server_name  test.relis.no;

        location / {
            proxy_pass http://localhost:8081;
            proxy_read_timeout 3600;
        }

        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header X-Forwarded-Proto https;

        ssl_certificate /etc/letsencrypt/live/test.relis.no/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/test.relis.no/privkey.pem;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_prefer_server_ciphers on;
        ssl_dhparam /etc/ssl/certs/dhparam.pem;

        ssl_ciphers ALL;
        ssl_session_timeout 1d;
        ssl_session_cache shared:SSL:50m;
        add_header Strict-Transport-Security max-age=15768000;
}

(ssl_ciphers ALL; is just for testing)

Adding debug connection information yields(this client is Safari on Ipad):

13409#0: *390 accept: 195.18.161.2 fd:3
 13409#0: posix_memalign: 00007F8FF0719A90:256 @16
 13409#0: *390 event timer add: 3: 60000:1463745705191
 13409#0: *390 reusable connection: 1
 13409#0: *390 epoll add event: fd:3 op:1 ev:80002001
 13409#0: *390 http check ssl handshake
 13409#0: *390 http recv(): 1
 13409#0: *390 https ssl handshake: 0x16
 13409#0: *390 SSL server name: "test.relis.no"
 13409#0: *390 SSL NPN advertised
 13409#0: *390 SSL_do_handshake: -1
 13409#0: *390 SSL_get_error: 2
 13409#0: *390 reusable connection: 0
 13409#0: *390 SSL handshake handler: 0
 13409#0: *390 SSL_do_handshake: -1
 13409#0: *390 SSL_get_error: 5
  2016/05/20 14:00:45 [info] 13409#0: *390 peer closed connection in SSL handshake (104: Connection reset by peer) while SSL handshaking, client: 195.18.161.2, server: 0.0.0.0:443
 13409#0: *390 close http connection: 3
 13409#0: *390 SSL_shutdown: 1
 13409#0: *390 event timer del: 3: 1463745705191
 13409#0: *390 reusable connection: 0
 13409#0: *390 free: 00007F8FF0731FF0, unused: 3
 13409#0: *390 free: 00007F8FF0719A90, unused: 136
 13409#0: *391 accept: 195.18.161.2 fd:3
 13409#0: posix_memalign: 00007F8FF0719A90:256 @16
 13409#0: *391 event timer add: 3: 60000:1463745705259
 13409#0: *391 reusable connection: 1
 13409#0: *391 epoll add event: fd:3 op:1 ev:80002001
 13409#0: *391 http check ssl handshake
 13409#0: *391 http recv(): 1
 13409#0: *391 https ssl handshake: 0x16
 13409#0: *391 SSL_do_handshake: -1
 13409#0: *391 SSL_get_error: 1
  2016/05/20 14:00:45 [info] 13409#0: *391 SSL_do_handshake() failed (SSL: error:140A1175:SSL routines:SSL_BYTES_TO_CIPHER_LIST:inappropriate fallback) while SSL handshaking, client: 195.18.161.2, server: 0.0.0.0:443
 13409#0: *391 close http connection: 3
 13409#0: *391 SSL_shutdown: 1
 13409#0: *391 event timer del: 3: 1463745705259
 13409#0: *391 reusable connection: 0
 13409#0: *391 free: 00007F8FF0731FF0, unused: 16
 13409#0: *391 free: 00007F8FF0719A90, unused: 136
 13409#0: *392 accept: 195.18.161.2 fd:3
 13409#0: posix_memalign: 00007F8FF0719A90:256 @16
 13409#0: *392 event timer add: 3: 60000:1463745705310
 13409#0: *392 reusable connection: 1
 13409#0: *392 epoll add event: fd:3 op:1 ev:80002001
 13409#0: *392 http check ssl handshake
 13409#0: *392 http recv(): 1
 13409#0: *392 https ssl handshake: 0x16
 13409#0: *392 SSL_do_handshake: -1
 13409#0: *392 SSL_get_error: 1
  2016/05/20 14:00:45 [info] 13409#0: *392 SSL_do_handshake() failed (SSL: error:1408A10B:SSL routines:SSL3_GET_CLIENT_HELLO:wrong version number) while SSL handshaking, client: 195.18.161.2, server: 0.0.0.0:443
 13409#0: *392 close http connection: 3
 13409#0: *392 SSL_shutdown: 1
 13409#0: *392 event timer del: 3: 1463745705310
 13409#0: *392 reusable connection: 0
 13409#0: *392 free: 00007F8FF0731FF0, unused: 16
 13409#0: *392 free: 00007F8FF0719A90, unused: 136

OpenSSL(version 1.0.1e-fips 11 Feb 2013) ciphers prints:

ECDHE-RSA-AES256-GCM-SHA384
ECDHE-ECDSA-AES256-GCM-SHA384
ECDHE-RSA-AES256-SHA384
ECDHE-ECDSA-AES256-SHA384
ECDHE-RSA-AES256-SHA
ECDHE-ECDSA-AES256-SHA
DHE-DSS-AES256-GCM-SHA384
DHE-RSA-AES256-GCM-SHA384
DHE-RSA-AES256-SHA256
DHE-DSS-AES256-SHA256
DHE-RSA-AES256-SHA
DHE-DSS-AES256-SHA
DHE-RSA-CAMELLIA256-SHA
DHE-DSS-CAMELLIA256-SHA
ECDH-RSA-AES256-GCM-SHA384
ECDH-ECDSA-AES256-GCM-SHA384
ECDH-RSA-AES256-SHA384
ECDH-ECDSA-AES256-SHA384
ECDH-RSA-AES256-SHA
ECDH-ECDSA-AES256-SHA
AES256-GCM-SHA384
AES256-SHA256
AES256-SHA
CAMELLIA256-SHA
PSK-AES256-CBC-SHA
ECDHE-RSA-AES128-GCM-SHA256
ECDHE-ECDSA-AES128-GCM-SHA256
ECDHE-RSA-AES128-SHA256
ECDHE-ECDSA-AES128-SHA256
ECDHE-RSA-AES128-SHA
ECDHE-ECDSA-AES128-SHA
DHE-DSS-AES128-GCM-SHA256
DHE-RSA-AES128-GCM-SHA256
DHE-RSA-AES128-SHA256
DHE-DSS-AES128-SHA256
DHE-RSA-AES128-SHA
DHE-DSS-AES128-SHA
ECDHE-RSA-DES-CBC3-SHA
ECDHE-ECDSA-DES-CBC3-SHA
DHE-RSA-SEED-SHA
DHE-DSS-SEED-SHA
DHE-RSA-CAMELLIA128-SHA
DHE-DSS-CAMELLIA128-SHA
EDH-RSA-DES-CBC3-SHA
EDH-DSS-DES-CBC3-SHA
ECDH-RSA-AES128-GCM-SHA256
ECDH-ECDSA-AES128-GCM-SHA256
ECDH-RSA-AES128-SHA256
ECDH-ECDSA-AES128-SHA256
ECDH-RSA-AES128-SHA
ECDH-ECDSA-AES128-SHA
ECDH-RSA-DES-CBC3-SHA
ECDH-ECDSA-DES-CBC3-SHA
AES128-GCM-SHA256
AES128-SHA256
AES128-SHA
SEED-SHA
CAMELLIA128-SHA
DES-CBC3-SHA
IDEA-CBC-SHA
PSK-AES128-CBC-SHA
PSK-3DES-EDE-CBC-SHA
KRB5-IDEA-CBC-SHA
KRB5-DES-CBC3-SHA
KRB5-IDEA-CBC-MD5
KRB5-DES-CBC3-MD5
ECDHE-RSA-RC4-SHA
ECDHE-ECDSA-RC4-SHA
ECDH-RSA-RC4-SHA
ECDH-ECDSA-RC4-SHA
RC4-SHA
RC4-MD5
PSK-RC4-SHA
KRB5-RC4-SHA
KRB5-RC4-MD5

ldd which nginx | grep ssl: libssl.so.10 => /lib64/libssl.so.10 (0x00007f413c860000)

nginx -V:

nginx version: nginx/1.6.3
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) 
TLS SNI support enabled
configure arguments: --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/run/nginx.pid --lock-path=/run/lock/subsys/nginx --user=nginx --group=nginx --with-file-aio --with-ipv6 --with-http_ssl_module --with-http_spdy_module --with-http_realip_module --with-http_addition_module --with-http_xslt_module --with-http_image_filter_module --with-http_geoip_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_degradation_module --with-http_stub_status_module --with-http_perl_module --with-mail --with-mail_ssl_module --with-pcre --with-pcre-jit --with-google_perftools_module --with-debug --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' --with-ld-opt='-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,-E'

Solution 1

It looks to me like any client which tries to use TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 fails. This is your most preferred cipher but all the clients that succeed do not support it so skip past it and all those that fail do support it.

I'm not sure why this is to be honest - maybe a bad implementation of TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 in whatever SSL library you are using (openssl?)? Not aware of any issues that would cause this but at same time this is not usually a cipher people would pick. If your SSL software supports SHA256 then it must be pretty modern and there's usually better ciphers you'd want over this one.

The other interesting thing to note is that the better ciphers are NOT listed - even though you have set ssl_ciphers to ALL. That is very odd! It would suggest to me you are on an old version of your SSL software but then why would it offer TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 at all (though it seems to me even when it does it doesn't support it)? At the same time it does support TLSv1.2 and successfully uses TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA for that (at least according to ssllabs) suggesting it's a more recent library. Very odd indeed.

Which SSL library are you using and what version? That might clear things up a little.

However in the meantime if you disable TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 I'd imagine this will work. You can use the Mozilla SSL configuration tool to get the settings you need, but these are the settings I use to have good support even for most older browsers but use the newer ciphers first if client supports them:

ssl_ciphers 'EECDH+AES128:EECDH+AES256:+SHA:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:RSA+3DES:!DSS';

Note your version of your SSL library may not support all of these ciphers.

Solution 2

I know this is an old post but in case anyone else runs into this issue or a similar SSL issue I found after a days worth of troubleshooting that it was related to my nginx version.

I was on nginx/1.6.3 and updated to current nginx/1.14.0 and the errors were solved.

I believe this was due to the older version of nginx using an older openSSL version which caused it not to properly read my newer SSL Certificate.

Solution 3

I found this, which made me test openssl s_client -connect with -tls1/-tls1_1/-tls1_2.

-tls1 and tls1_1 worked, but -tls1_2 failed as before with error 104.

I should probably dig further with this new information, but I am so fed up right now. So since it seems to work I will leave it for now.

Too bad the ssllabs score now is capped at C because of lacking TLS1.2, though.

Comments

[Admin]

Thank you for your answer. THe SSL library is OpenSSL 1.0.1e-fips. I tried the Mozilla configuration tool before posting, same result. I'll have to keep debugging.

[Admin]

Tried forcing protocol with -tls1, -tls1_1 and -tls1_2. openssl s_client -connect test.relis.no:443 -tls1_2 gives: CONNECTED(00000003) 140521595524768:error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number:s3_pkt.c:339:

[Barry Pollard]

I would guess that Nginx was built with older version of openssl and was updated on system after without rebuilding Nginx?