Weird interaction with systemctl with Haproxy on CentOS 7

Question

Running haproxy through systemctl is different than running it manually.

• on manual start, everything works.
• starting through systemctl, haproxy is not able to find the service it proxies, and answers with a 503.

Here's the output when starting through systemctl:

> sudo systemctl status  -l haproxy.service
haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled)
                Active: active (running) since Wed 2014-12-24 08:08:49 EST; 4min 59s ago
                Main PID: 20307 (haproxy)
                CGroup: /system.slice/haproxy.service
                └─20307 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid

                Dec 24 08:08:49 localhost.localdomain systemd[1]: Starting HAProxy Load Balancer...
                Dec 24 08:08:49 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.

Like this, haproxy just says 503 for my backend.

If I now stop the haproxy service, and instead run it manually on the command line, like so:

sudo /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid

it proxies fine to my server.

I've installed haproxy through yum, and I'm using an exceedingly simple configuration:

# defaults above

frontend  main *:80
    default_backend app

backend app
    option httpclose
    option forwardfor
    server web1 127.0.0.1:8001

Any idea what the difference could be? I've tried running it both daemonized and not.

This is the log when running with systemctl:

Proxy app started.
127.0.0.1:44391 [24/Dec/2014:09:01:42.403] main app/web1 0/0/-1/-1/0 503 212 - - SC-- 0/0/0/0/3 0/0 "GET / HTTP/1.1"

This is the log when running manually:

Proxy app started.
127.0.0.1:44393 [24/Dec/2014:09:02:11.758] main app/web1 0/0/0/2/2 200 5699 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1"

To be clear, I can successfully curl 127.0.0.1:8001 as both my deploy user, with sudo and the haproxy user.

Edit with more info

I tried using nginx as a proxy instead, to no avail. So it is not haproxy that is at fault here. nginx says 502 Bad Gateway, and the logs says:

2014/12/24 16:57:04 [crit] 23214#0: *1 connect() to 127.0.0.1:8001 failed (13: Permission denied) while connecting to upstream, client: 192.168.34.1, server: www.zombieclj.local, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8001/", host: "www.zombieclj.local"

So yeah, there is some security thing going on. I am still able to curl to 127.0.0.1:8001. Stopping the firewall doesn't change anything.

Solution 1

The problem was SELinux only allowing the web server to make outbound connections to a limited set of ports.

Fixed by doing:

semanage port --add --type http_port_t --proto tcp 8001

after installing semanage with

yum install policycoreutils-python

Nodejs Nginx error: (13: Permission denied) while connecting to upstream

Solution 2

I used the Audit2Allow function found on the centos wiki

Review the rule:

grep haproxy /var/log/audit/audit.log | audit2allow -m haproxy

Create a policy file (*.pp).

grep haproxy /var/log/audit/audit.log | audit2allow -M haproxy

now when installing the package I also include the haproxy.pp file and load it to selinux with

semodule -i haproxy.pp

Comments

[JasonAzze]

When you start haproxy manually, what user account do you use?

[Michael Hampton]

What did haproxy log?

[Magnar]

@JasonAzze I run it with sudo.

[Magnar]

@MichaelHampton I've only seen it log something along the lines of "Proxy app started", which isn't very useful.

[Sombriks]

This solution solved my problem. it should be marked as solution.

[Vijay Katam]

I ran into this issue when installing haproxy on a stock RHEL 7 image, for me the the port I wanted to use "8000" was already taken up by soundd_port_t so my solution involved some research using the Audit2Allow answer below.

[Vijay Katam]

The port I was running the backend service on "8000" was already taken by something called soundd_port_t, I used the solution documented at mankier.com/8/haproxy_selinux#Booleans which is to allow haproxy to connect to any tcp port.