Gitlab behind Haproxy(SSL)

Question

we have a virtualized server (esxi) with the typical configuration:

[Client] https -> [pfsense -> haproxy] - http -> [vm]

And now I am trying to configure a new virtual server with gitlab, and I can not find the correct configuration, inside the private network gitlab works correctly, but when I try to access from outside haproxy responds with a 503 error. After reading and trying several Configurations I can not make it work, I am sure that it is nginx problem (or so I think), since if on the same server I install apache (just to test) that server works correctly from outside.

The target is something like this:

[Client] https -> [pfsense -> haproxy] - http -> [gitlab]

Pfsense has ipen ports 80 and 443(I'm not sure if we need to open another to ssh or unicorn)

Some configurations:

gitlab.rc
external_url 'https://mydomain.extension'

Haproxy(works fine to others vms)

global
maxconn         10000
stats socket /tmp/haproxy.socket level admin
uid         80
gid         80
nbproc          1
chroot          /tmp/haproxy_chroot
daemon
tune.ssl.default-dh-param   2048
server-state-file /tmp/haproxy_server_state
ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK

frontend http_redirectTo_https
    bind            publicIP:80 name publicIP:80   
    mode            http
    log         global
    option          httpclose
    option          forwardfor
    acl https ssl_fc
    http-request set-header     X-Forwarded-Proto http if !https
    http-request set-header     X-Forwarded-Proto https if https
    timeout client      30000
    redirect scheme https code 301 if !{ ssl_fc }

frontend https_input
    bind            publicIP:443 name publicIP:443 ssl ssl  crt /certs/certific.pem no-sslv3 crt /var/etc/haproxy/https_frontend.pem  
    mode            http
    log         global
    option          http-keep-alive
    timeout client      30000
    acl         aclAdm  hdr(host) -i adm.domain.ext
    acl         aclOne  hdr(host) -i one.domain.ext
    acl         aclTwo  hdr(host) -i two.domain.ext
    acl         aclGit  hdr(host) -i git.domain.ext
    use_backend adm_backend_http_ipvANY  if  aclAdm 
    use_backend one_backend_http_ipvANY  if  aclOne 
    use_backend two_backend_http_ipvANY  if  aclTwo
    use_backend git_backend_http_ipvANY  if  aclGit

I remove the others backends

backend git_backend_http_ipvANY
    mode            http
    log         global
    balance         leastconn
    timeout connect     30000
    timeout server      30000
    retries         3
    option          httpchk OPTIONS / 
    option forwardfor
    option http-server-close
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    server          gitServer private_ip:80 check inter 1000

I guess the problem is with nginx, thanks!

Solution

The problem was, haproxy doesn't detect the server up so we suspect the health check configuration it's wrong:

option          httpchk OPTIONS / 

I just change it to basic check and haproxy detect correctly the server, and in fact, works fine.

Comments

[Admin]

Probably unrelated, but note this: check inter 100 ... that's configured to do a health check every 100 milliseconds... a bit aggressive. To troubleshoot HAProxy connections, reading the HAProxy log is the first thing you need to do.

[Admin]

Yes, it's 1000 not 100, copy error, thanks

[Admin]

Well, seeing the stats, haproxy shows the server like DOWN "Layer 7 wrong: Not found" but the server is running correctly, the logs doesn't show any error

[Admin]

Well, it appears to be returning a 404, whether it's logging it or not. Why are you using the argument OPTIONS / for option httpchk? That seems unusual.

[Admin]

You are COMPLETELY right, I really don't know why, I'm using the pfsense frontend to write the configuration, I changed the httpchk to basic in haproxy and everything works, it's possible that when haproxy check the server if haproxy don't detect, simply close the door? for me doesn't make sense the error only for the check protocol... Thanks a lot!

[Admin]

If I understand what you are asking, the explanation is this: If the health check fails for any reason -- whether that is due to misconfiguration, or because the backend host really is unhealthy, or network errors, etc. -- when that happens, the backend host is considered down, so no traffic would ever be sent to it by the proxy... and HAProxy would return 503 Service Unavailable for each request, by design, when that is the case.