HaProxy giving - 503 Service Unavailable

Question

I am using below configuring for performing loadbalancing for two Tomcat servers. And I configured HAProxy to perform SSL/TLS bridging/re-encryption.

#---------------------------------------------------------------------
# Example configuration for a possible web application.  See the
# full configuration options online.
#
#   http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local2 debug

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats
    ssl-server-verify none

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend  ft_main
default_backend bk_main
bind 192.168.192.175:443 ssl crt /home/testuser/Software/apache-tomcat-7.0.32/keystore/haproxy-cert.pem no-sslv3
mode http

#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend bk_main
    balance     roundrobin
    retries 3
server srv01 192.168.192.173:8443 weight 1 maxconn 100 check no-sslv3 ssl verify none 
server srv01 192.168.192.174:8443 weight 1 maxconn 100 check no-sslv3 ssl verify none 

When I am starting the service I am getting following error

Message from syslogd@localhost at Aug 17 17:06:12 ...
 haproxy[2593]: backend bk_main has no server available!

And I am observing following logs in /var/log/haproxy.log

Aug 17 16:52:07 localhost haproxy[2495]: backend bk_main has no server available!
Aug 17 16:52:25 localhost haproxy[2496]: 120.117.50.250:52088 [17/Aug/2015:16:52:25.084] ft_main~ bk_main/ 556/-1/-1/-1/556 503 212 - - SC-- 1/1/0/0/0 0/0 "GET /test/healcheck.jsp HTTP/1.1"
Aug 17 16:52:26 localhost haproxy[2496]: 120.117.50.250:52089 [17/Aug/2015:16:52:25.336] ft_main~ bk_main/ 1044/-1/-1/-1/1044 503 212 - - SC-- 0/0/0/0/0 0/0 "GET /test/healcheck.jsp HTTP/1.1"
Aug 17 16:52:27 localhost haproxy[2496]: 120.117.50.250:52090 [17/Aug/2015:16:52:27.371] ft_main~ bk_main/ 268/-1/-1/-1/268 503 212 - - SC-- 1/1/0/0/0 0/0 "GET /test/healcheck.jsp HTTP/1.1"
Aug 17 16:52:28 localhost haproxy[2496]: 120.117.50.250:52091 [17/Aug/2015:16:52:27.623] ft_main~ bk_main/ 671/-1/-1/-1/671 503 212 - - SC-- 0/0/0/0/0 0/0 "GET /test/healcheck.jsp HTTP/1.1"
Aug 17 16:52:29 localhost haproxy[2496]: 120.117.50.250:52092 [17/Aug/2015:16:52:29.395] ft_main~ bk_main/ 273/-1/-1/-1/273 503 212 - - SC-- 1/1/0/0/0 0/0 "GET /test/healcheck.jsp HTTP/1.1"
Aug 17 16:52:30 localhost haproxy[2496]: 120.117.50.250:52093 [17/Aug/2015:16:52:29.646] ft_main~ bk_main/ 701/-1/-1/-1/701 503 212 - - SC-- 0/0/0/0/0 0/0 "GET /test/healcheck.jsp HTTP/1.1"
Aug 17 16:52:51 localhost haproxy[2496]: 120.117.50.250:52094 [17/Aug/2015:16:52:51.268] ft_main~ bk_main/ 274/-1/-1/-1/274 503 212 - - SC-- 1/1/0/0/0 0/0 "GET /test/healcheck.jsp HTTP/1.1"
Aug 17 16:57:15 localhost haproxy[2496]: 192.168.192.168:54329 [17/Aug/2015:16:57:15.560] ft_main/1: SSL handshake failure
Aug 17 17:00:34 localhost haproxy[2538]: Proxy ft_main started.
Aug 17 17:00:34 localhost haproxy[2538]: Proxy bk_main started.
Aug 17 17:00:34 localhost haproxy[2538]: Server bk_main/srv01 is DOWN, reason: Layer6 invalid response, info: "SSL handshake failure", check duration: 30ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Aug 17 17:00:34 localhost haproxy[2538]: backend bk_main has no server available!
Aug 17 17:06:12 localhost haproxy[2593]: Proxy ft_main started.
Aug 17 17:06:12 localhost haproxy[2593]: Proxy bk_main started.
Aug 17 17:06:12 localhost haproxy[2593]: Server bk_main/srv01 is DOWN, reason: Layer6 invalid response, info: "SSL handshake failure", check duration: 25ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Aug 17 17:06:12 localhost haproxy[2593]: backend bk_main has no server available!

Based on the "SC" flags present in above flag my understand is that it failed to establish a server connection. So is it a handshake issue? I am using self signed certificate. And in server directive I mentioned "verify none" so what can be the issue?

Solution

Of course try the connection to backends yourself:

openssl s_client -connect 192.168.192.173:8443

This way you'll remove most of possible causes.

What if s_client works, but haproxy doesn't? On SELinux enforcing (for example CentOS 7 with default settings) exactly this happens and you need to explicitly allow haproxy to connect to any backend port:

setsebool -P haproxy_connect_any 1