HAProxy backend with TLS stopped working

Question

I'm having trouble with one of our HAProxy-Servers that uses a backend with TLS. When starting HAProxy the backend will report all servers as down:

Server web_remote/apache_rem_1 is DOWN, reason: Layer6 invalid response, info: "SSL handshake failure", check duration: 41ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

The configuration for the backend is as follows:

backend web_remote
    balance         leastconn
    option          httpchk         HEAD /
    option          redispatch
    retries         3

    default-server  inter 5000 rise 2 fall 5 maxconn 10000 maxqueue 50000

    server apache_rem_1  192.0.2.1:12345           check maxconn 1000 maxqueue 5000 ssl ca-file /etc/ssl/web.pem
    server apache_rem_2  2001:DB8:3:4:5:6:8:12345  check maxconn 1000 maxqueue 5000 ssl ca-file /etc/ssl/web.pem

This backend worked just fine until now, a quick wget on the server also worked and openssl s_client reports the certificate of the backend to be valid.

Is there anything else for HAProxy/TLS that could be configured wrong? How could I debug this issue when everything else reports the handshake was successful?

Solution

After trying to find ways to debug with HAProxy I finally gave in and used tcpdump to debug the network traffic.

By importing the TLS key and the PCAP file into my SSL-enabled Wireshark, I found the error very quickly: The Root CA was unknown to HAProxy, so the verify was failing. It turned out, a coworker changed the cert but didn't know this PEM needs the complete trust chain. Adding the correct root CA to the .PEM file fixed this and the server is back to normal operations.

Comments

[Izzy]

That is a default, so adding that did not make any difference.

[GregL]

As far as all the documentation tells me, check-ssl isn't a default. If you have something that says otherwise, I'd like to see it! What about changing the option httpchk HEAD / to option httpchk GET /?