HAProxy "option redispatch" not working when I put into maintenance mode

Question

I'm having an issue with HAProxy not redispatching connections to servers. When I put a server into maintenance mode none of the active connections are being disconnected/transferred to other active servers and its causing timeouts for my end users.

My conf is as follows:

    #---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local0

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     20000
    user        haproxy
    group       haproxy
    daemon
    tune.ssl.default-dh-param  2048

    #Specify only the strong ciphers when the LB is doing SSL/TLS termination
    ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESG
CM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
    ssl-default-server-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AE
SGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS


    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats level admin

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------

defaults
    mode                    tcp
    log                     global
    option                  logasap
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    option httpclose
    option abortonclose
    timeout http-request    4s    #How long should we wait?
    timeout queue           30s
    timeout connect         4s     #How long should just the connect take?
    timeout client          30s
    timeout server          30s
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 19500

#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------

frontend www-website-com
    bind 192.168.1.154:80
    bind 192.168.1.154:443
    option tcplog
    mode tcp
    default_backend             www-website-servers


#---------------------------------------------------------------------
# BackEnds Section
#---------------------------------------------------------------------

backend www-website-servers
    mode tcp
    balance leastconn
#    option ssl-hello-chk
    server sv1 192.168.1.1:443 check
    server sv2 192.168.1.3:443 check
    server sv3 192.168.1.6:443 check
    server sv4 192.168.1.9:443 check

The mode for all the front end nodes is TCP.

If anyone has any suggestions they would be greatly appreciated!

Thanks!

Solution

Well for starters, according to the HAProxy docs option redispatch only works for HTTP proxies:

In HTTP mode, if a server designated by a cookie is down, clients may definitely stick to it because they cannot flush the cookie, so they will not be able to access the service anymore.

Specifying "option redispatch" will allow the proxy to break their persistence and redistribute them to a working server.

It also allows to retry last connection to another server in case of multiple connection failures. Of course, it requires having "retries" set to a nonzero value.

This form is the preferred form, which replaces both the "redispatch" and "redisp" keywords.

If this option has been enabled in a "defaults" section, it can be disabled in a specific instance by prepending the "no" keyword before it.

What you might try is adding on-marked-down shutdown-sessions to your server lines in the backend.

backend www-website-servers
  mode tcp
  balance leastconn
#  option ssl-hello-chk
  server sv1 192.168.1.1:443 check on-marked-down shutdown-sessions
  server sv2 192.168.1.3:443 check on-marked-down shutdown-sessions
  server sv3 192.168.1.6:443 check on-marked-down shutdown-sessions
  server sv4 192.168.1.9:443 check on-marked-down shutdown-sessions

The docs say:

on-marked-down Modify what occurs when a server is marked down.
Currently one action is available:
- shutdown-sessions: Shutdown peer sessions. When this setting is enabled, all connections to the server are immediately terminated when the server goes down. It might be used if the health check detects more complex cases than a simple connection status, and long timeouts would cause the service to remain unresponsive for too long a time. For instance, a health check might detect that a database is stuck and that there's no chance to reuse existing connections anymore. Connections killed this way are logged with a 'D' termination code (for "Down").

Comments

[GregL]

Can you post the full config, including frontend and backend sections?

[user1857654]

@GregL I just updated it for you. Thanks again for any help!

[GregL]

Thanks, a couple of other questions. Are you putting them in Maintenance, or Drain? Are these long-living connections?

[user1857654]

@GregL I am putting them into Maintenance mode they even indicate a (MAINT) next to the servers when I put them in that mode.

[GregL]

How long are you waiting between setting MAINT and doing the work on the server? Does the 'Sessions - Cur' counter on the stats page show 0?

[user1857654]

@GregL I start work immediately and no the session current has active sessions. I was just assuming they would be instantly forced over. Was I mistaken? Is there a way to make it happen right away?

[Karl Morrison]

What OS. You can't not say.