HAproxy 503 Service Unavailable No server is available to handle this request

Question

I'm new to this load balancing scenario and I'm being tasked to figure out how to make this load balancing works.

My environment:

Centos 6.4 64 Bit
Webserver: Lighttpd
All running in ESXI
virtual IP: 192.168.1.6
LB1: 192.168.1.4
LB2: 192.168.1.5
Webserver 1: 192.168.1.12
Webserver 2: 192.168.1.13
Gateway: 192.168.1.1

Trying to run a test in the lab prior production with HAproxy and keepalived. Here is what I have on my keepalived setting:

! Configuration File for keepalived

global_defs {
   notification_email {
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 192.168.1.4
   smtp_connect_timeout 30
   router_id 192.168.1.1
}

vrrp_script chk_haproxy {
script "killall -0 haproxy"
interval 1                     # check every second
weight 2                       # add 2 points of prio if OK
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 101  #priority 101 for master
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.1.6

    }

    track_script {
    chk_haproxy
            }

            }

and here is my setting for HAproxy

#---------------------------------------------------------------------
# Example configuration for a possible web application.  See the
# full configuration options online.
#
#   http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend  main *:80
#    acl url_static       path_beg       -i /static /images /javascript /stylesheets
#    acl url_static       path_end       -i .jpg .gif .png .css .js

#    use_backend static          if url_static
#    default_backend             view

#---------------------------------------------------------------------
# static backend for serving up images, stylesheets and such
#---------------------------------------------------------------------
#backend static
#    balance     roundrobin
#    server      static 127.0.0.1:4331 check

#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend app
mode tcp
    balance     roundrobin
    server  server1 192.168.1.12:80 check inter 2000 rise 2 fall 5
    server  server2 192.168.1.13:80 check inter 2000 rise 2 fall 5

When I started the HAproxy, I got thiss error and I'm not quite sure where to start looking to fix it. Perhaps someone who has done this many times can help me shed some light?

503 Service Unavailable No server is available to handle this request. 

However manually connecting to the webserver1 and webserver2 works just fine.

All I want is just a simple load balancing for the webserver that sit behind the HAproxy. Any advise or suggestion is absolutely appreciated. Please help? Many thanks.

Solution 1

I've never used HAproxy but a quick search leads me to think you need to add default_backend app immediately below frontend main *:80. I see nowhere in that configuration connecting the backend and frontend together.

Solution 2

Problem is in your HAProxy configuration. When I remove all comments from your config, I will get this:

global
  log         127.0.0.1 local2

  chroot      /var/lib/haproxy
  pidfile     /var/run/haproxy.pid
  maxconn     4000
  user        haproxy
  group       haproxy
  daemon

  stats socket /var/lib/haproxy/stats

defaults
  mode                    http
  log                     global
  option                  httplog
  option                  dontlognull
  option http-server-close
  option forwardfor       except 127.0.0.0/8
  option                  redispatch
  retries                 3
  timeout http-request    10s
  timeout queue           1m
  timeout connect         10s
  timeout client          1m
  timeout server          1m
  timeout http-keep-alive 10s
  timeout check           10s
  maxconn                 3000

frontend  main *:80

backend app
  mode tcp
  balance roundrobin
  server  server1 192.168.1.12:80 check inter 2000 rise 2 fall 5
  server  server2 192.168.1.13:80 check inter 2000 rise 2 fall 5

And now you can clearly see that there is no configuration of frontend at all. Requests comes to HAProxy via frontend main but HAProxy doesn't know which servers are reliable to handle it, so will return 503.

You have to link backend to frontend with default_backend or with acl.

You should use stats too, not only with socket but with protected web interface too. I can shows you information about clusters behind haproxy, which servers are offline, which has any problems, about response times and so on. Very usefull for debugging.

Solution 3

I got a similar error because HAProxy thought my backend was down due to the default health check it does. I disabled the health check and the 503 went away.

I'm using pfsense GUI:

Solution 4

I had a slighty different configuration so this is not the answer to your specific problem, but it may help others experiencing the 503 error with HAProxy.

My Haproxy was configured like this:

use_backend be_external-service-1-0 if { hdr_beg(host) -i external-service-1-0 }

meaning that the backend will only be used if the host header starts with external-service-1-0.

In my case, the reason for the 503 error was that the client was sending the following request headers:

X-App-Id: 98d77fae1082342342323423423452ae203489234

Host: external-service-1-0.prod-drb-external.svc.cluster.local:8080 Connection: Keep-Alive

Note that extra line between X-App-Id and Host. An empty line makes HAProxy believe that this is the end of the HTTP Headers, therefore it ignored the Host header and it could not find the correct backend.

Solution 5

In my case the configuration seemed proper, but I was still getting the error. The fix was simply allowing haproxy to run with selinux, or disabling selinux entirely on a test VM.

setsebool -P haproxy_connect_any 1
setenforce 0

View more solutions

Comments

[Le Dude]

Thanks @yoonix. Make sense and I tried it by uncomment the default_backend and change the view to app. and change the mode to http since there's an error that told me that I have to change it. Restarting haproxy is now cleaner but now I got different error. 404 - Not Found

[Admin]

I would think that error is actually coming from the backend, not HAproxy. Check the request and error logs for the webservers on the backend.

[Le Dude]

You beat me to the punch. I check the lighttpd configuration and the vhost config was shown wrong. For example, I'm trying to connect to testlab.mydomain.com associated with haproxy VIP. However the lighttpd vhosts configuration was pointed to mylab.mydomain.com. Thus it couldn't find it and error 404 occured. Many thanks for the advise yoonix. I'll consider this as answered. :-) Have a wonderful day or evening where ever you are. :-)

[HappyCoder]

How did you disable health check, I think I am having the same issue but not 100% sure how to disable it.

[Charlie]

HappyCoder I updated with a screenshot of pfsense, which is what I'm using.

[HappyCoder]

Thank you! I found the issue, when updating the app I broke a route to a page that returns OK. haproxy is configured to test this route every so often to ensure uptime... I should have checked this first...

[Eric Johnson]

Thank you! This fixed an issue I was fighting for a few hours with a SVN server.