Multiple processes listening on the same port; how is it possible?

Question

Multiple processes are listening on same port. But as far as I know only one process can listen on a port. Is it possible (how?) that multiple processes can listen on same port?

$ sudo lsof -n -i :80 | grep LISTEN
haproxy 2039 root    4u  IPv4  12874      0t0  TCP *:http (LISTEN)
haproxy 2042 root    4u  IPv4  12898      0t0  TCP *:http (LISTEN)
haproxy 2045 root    4u  IPv4  12923      0t0  TCP *:http (LISTEN)

pstree output:

init
  ├─acpid -c /etc/acpi/events -s /var/run/acpid.socket
  ├─atd
  ├─cron
  ├─dbus-daemon --system --fork
  ├─dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0 
  ├─docker -d
  │   └─6*[{docker}]
  ├─getty -8 38400 tty4
  ├─getty -8 38400 tty5
  ├─getty -8 38400 tty2
  ├─getty -8 38400 tty3
  ├─getty -8 38400 tty6
  ├─getty -8 38400 tty1
  ├─getty -8 38400 ttyS0
  ├─haproxy -f /etc/haproxy/haproxy.cfg
  ├─haproxy -f /etc/haproxy/haproxy.cfg
  ├─haproxy -f /etc/haproxy/haproxy.cfg

haproxy config:

global
    log /dev/log    local0
    log /dev/log    local1 notice
    chroot /var/lib/haproxy
    user ubuntu
    group ubuntu
    daemon 

defaults
    log global
    mode    http
    option  httplog
    option  dontlognull
        contimeout 5000
        clitimeout 50000
        srvtimeout 50000

listen appname 0.0.0.0:80
    mode http
    stats enable
    stats uri /haproxy?stats
    balance roundrobin
    option httpclose
    option forwardfor
    server lamp1 172.31.20.0:81 check
    server lamp2 172.31.20.1:81 check

Solution

It is possible. The goal is to process multiple incoming connections in parallel. Multiple haproxy instances may utilize separate CPU cores and work (semi-)independently. Incoming connection will be passed to idle haproxy (if available) instead of being queued to busy one.

I guess haproxy uses SO_REUSEPORT. man 7 socket explains this option like this:

SO_REUSEPORT (since Linux 3.9)

Permits multiple AF_INET or AF_INET6 sockets to be bound to an identical socket address. This option must be set on each socket (including the first socket) prior to calling bind(2) on the socket. To prevent port hijacking, all of the processes binding to the same address must have the same effective UID. This option can be employed with both TCP and UDP sockets.

For TCP sockets, this option allows accept(2) load distribution in a multithreaded server to be improved by using a distinct listener socket for each thread. This provides improved load distribution as compared to traditional techniques such using a single accept(2)-ing thread that distributes connections, or having multiple threads that compete to accept(2) from the same socket.

Also check SO_ATTACH_REUSEPORT_CBPF and SO_ATTACH_REUSEPORT_EBPF there.

---

Edit: I found this article (dated May 3, 2017); it seems to support my guess:

In the mean time, a new and much better SO_REUSEPORT implementation was brought to Linux kernel 3.9, allowing the load to be intelligently spread over multiple sockets. HAProxy could immediately benefit from this new improvement.

But it came with a problem [...]

Don't worry about the problem. The article describes workarounds and a solution. You may find it interesting if you're into this kind of stuff.

Comments

[Jaroslav Kucera]

What's your haproxy configuration? Do you server multiple domains?

[Kundan]

added haproxy config

[Jaroslav Kucera]

How many IPs do you have active (including localhost)? If 3, thats probably the reason.

[Kundan]

This is not related to ip. I was able to run 10 instance of haproxy.

[David Schwartz]

"But as far as I know only one process can listen on a port." I'm not sure why you would think that, but it's certainly not true. For the most obvious way to see how that can't be right -- suppose a process is listening on a port and then it calls fork. If only one process can listen on that port -- which one would it be?

[Kundan]

fork will create a new child process. But as I can see from pstree these are not child processes.