How to debug linux networking: Address already in use

Question

I have a Slackware linux box where I cannot start any service that listens on one particular port on localhost. By using strace I found out that the error happens on the bind() call, and the error is EADDRINUSE (Address already in use):

bind(3, {sa_family=AF_INET, sin_port=htons(874), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EADDRINUSE (Address already in use)

This happens with any process I try to start listening on that port, so it is not related to the process itself. The above strace output comes from the command strace -ff nc -l -p 874 -s 127.0.0.1.

So, this suggests there is a process already listening on localhost port 874. However, I can't seem to find it. The following commands all return nothing:

netstat -aplunt | grep :874
netstat -na | grep :874
lsof -i :874
lsof -i tcp | grep 874
fuser 874/tcp
socklist | grep 874
iptables -t filter -S | grep 874
iptables -t nat -S | grep 874
iptables -t mangle -S | grep 874
conntrack -L | grep 874

If I try to listen on 0.0.0.0:874 it fails with the same error. Listening on one of the IP addresses configured on a nic works OK, and listening to 127.0.0.2:874 also works OK. Listening on a different port works fine, also on 127.0.0.1 or 0.0.0.0.

So, now I am curious. How can I find out why the network stack returns EADDRINUSE here? What other things could I look at, or what other commands can I run to get more information?

Additional info:

• Kernel 4.1.31.
• Selinux is not used here.
• Trying to connect to 127.0.0.1 with telnet returns "Connection refused"
• I'm running the commands as root

Solution

If your host is an NFS client, it may be using source port 874 for an NFS mount. I suspect that because the connection does not originate from userspace it may not be visible to the tools you've used so far.

Consider one of the following:

• Adjust the sysctls sunrpc.min_resvport and sunrpc.max_resvport (default 665 and 1023) to change the range of source ports that the NFS client uses
• Use a listening port outside of this range
• Use the noresvport option on the NFS mount to use the non-privileged range (may have security implications)

Comments

[Admin]

Can you check if there is no established sessions on that port? Run netstat -aplunt | grep 874, just add 'a' flag. Or maybe some session using this port as the source port?(Although highly unlikely). But worth giving a shot.

[Admin]

No output. I've updated the netstat command in the question.

[Admin]

Is the port mentioned anywhere in iptables -S output?

[Admin]

I think @hertitu is right, it may be port forwarding in iptables. Also use nmap to check 874 tcp and udp port. What ip has localhost in /etc/hosts file? Check all configs in /etc and /usr/local/etc folders that they have 874 substring.

[Admin]

@hertitu how would portforwarding affect listening on a port? Anyway, I checked, and there are no iptables rules for that port. To be extra sure I flushed the firewall: no change.

[Admin]

Can you please also print output of strace -ff nc -l 874 also, The one you used is trying to make connections with 874 as source port. Thanks!

[Admin]

@AnirudhMalhotra I'm not making a connection, I'm listening on port 874, which is where the problem is. The command you showed listens on a random high port. There is no problem doing that.

[Admin]

Well here is what the man page of nc states: -p source_port Specifies the source port nc should use, subject to privilege restrictions and availability. -l Used to specify that nc should listen for an incoming connection rather than initiate a connection to a remote host. It is an error to use this option in conjunction with the -p, -s, or -z options. Additionally, any timeouts specified with the -w option are ignored. you cannot these flags in conjunction. So I will correct myself the command you used is rather contradictory. Thanks!

[Admin]

AFAIK Linux requires root privileges when listening to a port < 1000. Maybe that's the problem here.

[Admin]

Anyway, I have the same problem if I use a different process to listen: rsync --daemon --address=127.0.01 --port=874

[Admin]

I'm running all commands as root. Question updated.

[Admin]

finally pheww... TCPdump(ughh, should have thought about earlier). Glad it worked though.

[Admin]

Glad I could help, posted as an answer with some ways of mitigating the issue. P.S. On my machine the NFS connections are visible in the netstat -na output, albeit without associated processes (because they come from the kernel). They can't be seen with commands like lsof or fuser.