Haproxy tuning for performance?

Question

We are trying to find the best tuning options for haproxy for get and post request that come from a client (not users browsing the web type of deal).

Running a jmeter test with 30k threads that consists of 5 calls to the servers, 1 user reg, and a few update calls. These push json data though the pipeline.

Here us our current config for haproxy

global
        log /dev/log local0 #notice
        maxconn 14000
        tune.bufsize 128000
        user netcom
        group netcom
        pidfile /tmp/haproxy.pid
        daemon
        nbproc 7
        #debug
        #quiet

defaults
        log global
        mode http
        ### Options ###
        option httplog
        #option logasap
        option dontlog-normal
        #option dontlognull
        option redispatch
        option httpchk GET /?method=echo HTTP/1.1
        option tcp-smart-accept
        option tcp-smart-connect
        option http-server-close
        #option httpclose
        #option forceclose
        ### load balance strategy ###
        balance leastconn
        #balance roundrobin
        ### Other ###
        retries 5
        maxconn 14000
        backlog 100000
        ### Timeouts ###
        #timeout client          25s
        timeout client          60s
        #timeout connect          5s
        timeout connect         60s
        #timeout server          25s
        timeout server          60s
        timeout tunnel        3600s
        timeout http-keep-alive  1s
        #timeout http-request    15s
        timeout http-request    60s
        #timeout queue           30s
        timeout queue           30s
        timeout tarpit          60s

listen stats *:1212
        stats enable
        stats show-node
        stats show-desc xxxxProxy
        stats realm  xxxxProxy\ Statistics
        stats auth   xxxx:xxxx
        stats refresh 5s
        stats uri /

frontend http-in
        bind *:1111
        bind *:2222 ssl crt /home/netcom/nas/haproxy/xxxx.co.pem verify optional
        acl user_request url_reg method=user.register
        use_backend user_group if user_request
        default_backend other_group

backend user_group
        server n15 xxxx:8080 maxconn 3500 check port 8097 inter 2000
        server n2 xxxx:8080 maxconn 3500 check port 8097 inter 2000
        server n9 xxxx:8080 maxconn 3500 check port 8097 inter 2000
        server n14 xxxx:8080 maxconn 3500 check port 8097 inter 2000
        server n22 xxxx:8080 maxconn 3500 check port 8097 inter 2000
        server n24 xxxx:8080 maxconn 3500 check port 8097 inter 2000
        server n25 xxxx:8080 maxconn 3500 check port 8097 inter 2000

and our sysctl on centOS 6

net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_synack_retries = 2
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_tw_recycle = 1
net.core.wmem_max = 12582912
net.core.rmem_max = 12582912
net.ipv4.tcp_rmem = 20480 174760 25165824
net.ipv4.tcp_wmem = 20480 174760 25165824
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_no_metrics_save = 1
net.core.netdev_max_backlog = 10000
# Syn flood
net.ipv4.tcp_max_syn_backlog = 8096
net.core.somaxconn = 8096

anyone point out any blaring issues that they can see off the top of your head. Unfortunately I do not have the expertise in haproxy so looking for help from the community.

What I also prob need to figure out is how to find the max connections the box can handle, its on 1 gig network and all the backends are on one gig as well. Here is screen shot from the haproxy admin http://grab.by/r12c, note we are running it with more than one core so this is a snapshot of the one core.. since the web admin as far as I can tell cant show everything.. any idea how to get the max conn that haproxy is getting from cmd line?

anyhow just working though this and hope that anyone can give some tips or pointers.

Solution

Well the first thing is that it doesn't seem like you should be running multiple processes of haproxy. Typically you won't want to do that, especially because you are busy testing and trying to see the maxconn's. On a single core haproxy can way outperform the maxconn setting you have anyway.

I went through Snapt's sysctl's and you have most of the; I noticed it's also adding these --

    net.ipv4.tcp_tw_reuse = 1
    net.ipv4.tcp_fin_timeout = 30

Also, leastconn is not going to be worthwhile I would suggest roundrobin. Because you are doing HTTP traffic which consists of many small requests (I guess that depends though to be honest). These are such minor things though.