HAProxy configuration for RabbitMQ

Question

I have an API that sends messages to RabbitMQ.

I have a high availability RabbitMQ cluster behind HAProxy.

When I load test my API I start seeing lots of this:

Recovering from a network failure... Exception in the reader loop: AMQ::Protocol::EmptyResponseError: Empty response received from the server.

In my unicorn log.

If I connect directly to RabbitMQ as a posed to via haproxy I do not. Where am I going wrong, my haproxy config looks like this:

global
  log 127.0.0.1   local0
  log 127.0.0.1   local1 notice
  #log loghost    local0 info
  maxconn 4096
  #debug
  #quiet
  user haproxy
  group haproxy

defaults
  log     global
  mode    http
  retries 3
  timeout client 50s
  timeout connect 10s
  timeout server 50s
  option dontlognull
  option forwardfor
  option httplog
  option redispatch
  balance  roundrobin

# Set up application listeners here.

listen http_frontend
  bind *:80
  mode http
  default_backend http_backend
  option httpclose
  reqadd X-Forwarded-Proto:\ http

listen https_frontend
  bind *:443 ssl crt /etc/haproxy.pem
  mode http
  default_backend http_backend
  reqadd X-Forwarded-Proto:\ https

listen http_bucky_frontend
  bind *:1880
  mode http
  default_backend http_bucky_backend
  option httpclose
  reqadd X-Forwarded-Proto:\ http

listen https_bucky_frontend
  bind *:1443 ssl crt /etc/haproxy.pem
  mode http
  default_backend http_bucky_backend
  reqadd X-Forwarded-Proto:\ https

listen rabbitmq_frontend
  bind *:5672
  mode tcp
  default_backend rabbitmq_backend
  option tcplog

listen admin
  bind 127.0.0.1:22002
  mode http
  stats uri /


backend http_backend
  mode http
  server 0-http_backend x.x.x.x:9000 maxconn 100 check
  server 1-http_backend x.x.x.x:9000 maxconn 100 check

backend http_bucky_backend
  mode http
  option httpchk GET /status
  http-check expect string up
  server 0-http_bucky_backend x.x.x.x:9000 maxconn 100 check
  server 1-http_bucky_backend x.x.x.x:9000 maxconn 100 check

backend rabbitmq_backend
  balance roundrobin
  mode tcp
  server 0-rabbitmq_backend x.x.x.x:5672 maxconn 4000 check
  server 1-rabbitmq_backend x.x.x.x:5672 maxconn 4000 check

When under load the load balancer is normally 20-30% cpu

Solution 1

Nerijus is right, this issue is caused by the HAProxy having a client timeout, which means, if the connection is considered idle for more than X ms then the connection is dropped.

TCP can send keep-alive packets to ensure an idle connection should stay open.

You can check your TCP parameter for keep-alive packets using the following command:

$ cat /proc/sys/net/ipv4/tcp_keepalive_time

By default, this configuration is equal to 7200 seconds, which means TCP will begin to send keep-alive packets only after a connection is idle for more than 2 hours.

So, just update your HAProxy client timeout value to something > 2 hours, e.g:

timeout client 3h

And add the clitcpka option to your backend:

backend rabbitmq_backend
  balance roundrobin
  mode tcp
  option          clitcpka
  server 0-rabbitmq_backend x.x.x.x:5672 maxconn 4000 check
  server 1-rabbitmq_backend x.x.x.x:5672 maxconn 4000 check

Solution 2

Remvoe timeout client 50s from default section and move to all others except from rabbitmq_frontend. Client timeout means that it closes TCP session between rabbitmq client and haproxy so reading from this socket gets "".

Comments

[Timothy c]

clitcpka is not valid in a backend. It's a frontend config option