Configure HAProxy on Linux to accept 10K simultaneous connections

Question

I have HAProxy 1.5 running on Ubuntu 14.04 (modified). It accepts connections on http and https ports. Two backend applications process requests using persistent connection.

When I create around 2200 client connections haproxy stops accepting additional connections. But I want this system to accept at least 10K simultaneous connections.

Here is connection statistics:

# ss -s
TCP:   4119 (estab 4098, closed 6, orphaned 0, synrecv 0, timewait 6/0), ports 0

Transport Total     IP        IPv6
TCP       4113      4106      7
INET      4128      4117      11

I have already tuned maximum number of opened files for the process:

# cat /proc/1012/limits
Limit                     Soft Limit           Hard Limit           Units
Max open files            240017               240017               files

My haproxy.config file:

global
    log /dev/log syslog debug
    daemon
    user haproxy
    group haproxy
    maxconn 120000
    spread-checks 4

defaults
    log global
    timeout connect 30000ms
    timeout client 300000ms
    timeout server 300000ms

frontend http-in
    mode http
    bind :80
    option httplog
    option forwardfor
    reqadd X-Forwarded-Proto:\ http
    default_backend http-routers

frontend https-in
    mode http
    bind :443 ssl crt /opt/haproxy/cert.pem no-sslv3
    option httplog
    option forwardfor
    option http-server-close
    reqadd X-Forwarded-Proto:\ https
    default_backend http-routers

frontend ssl-in
    mode tcp
    bind :4443 ssl crt /opt/haproxy/cert.pem no-sslv3
    default_backend tcp-routers

backend http-routers
    mode http
    balance roundrobin
        server node0 192.168.10.2:80 check inter 1000
        server node1 192.168.10.2:80 check inter 1000

backend tcp-routers
    mode tcp
    balance roundrobin
        server node0 192.168.10.2:80 check inter 1000
        server node1 192.168.10.2:80 check inter 1000

Solution

As far as I know, listen block maxconn is different then global maxconn. With global maxconn you limit the max number of connections you let the haproxy process handle.

Listen / frontend section has its own maxconn, which limits the nubmer of connections per listener. So, try to set up maxconn in your frontend sections too, or at least set it up in default section.

So either:

 defaults
     maxconn 10000

or set it up per frontend.

Comments

[Roger Sherman]

Please post your haproxy.cfg.

[lexsys]

I have added my configuration to the post

[Clément Perroud]

Did you try to increase somaxconn just in case you spawn connections too fast?

[lexsys]

I can't established a single additional connection when the 2.2K limit is reached, so I think connection rate is not limiting factor. I tried to increased net.core.somaxconn = 1024, but it didn't work.

[lexsys]

Thank you, Jakov! This solution helped me - now I have 16K connections established.