netstat shows a listening port with no pid but lsof does not

Question

This question is similar to Network port open, but no process attached?

I've tried everything from there, reviewed the logs, etc... and can't find anything.

My netstat shows a TCP listening port and a UDP port without a pid. When I search lsof for those ports nothing comes up.

netstat -lntup
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:44231           0.0.0.0:*               LISTEN      -               
udp        0      0 0.0.0.0:55234           0.0.0.0:*                           - 

The following commands display nothing:

lsof | grep 44231
lsof | greo 55234
fuser -n tcp 44231
fuser -n udp 55234

After rebooting, those "same" two connections are there except with new port numbers:

netstat -lntup
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:45082           0.0.0.0:*               LISTEN      -               
udp        0      0 0.0.0.0:37398           0.0.0.0:*                           - 

And once again, the lsof and fuser commands show nothing.

Any ideas what they are? Should I be concerned about them?

Solution 1

From data you provided I'd say it's related to some NFS mounts or something using RPC.

you can check with rpcinfo -p for ports that might be used by some of RPC related services.

Here is how it looks on my system

# netstat -nlp | awk '{if ($NF == "-")print $0}'
tcp        0      0 0.0.0.0:55349           0.0.0.0:*               LISTEN      -               
udp        0      0 0.0.0.0:18049           0.0.0.0:*                           - 

# rpcinfo -p
   program vers proto   port
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp  10249  status
    100024    1   tcp  10249  status
    100021    1   udp  18049  nlockmgr
    100021    3   udp  18049  nlockmgr
    100021    4   udp  18049  nlockmgr
    100021    1   tcp  55349  nlockmgr
    100021    3   tcp  55349  nlockmgr
    100021    4   tcp  55349  nlockmgr

Solution 2

Some processes/pids are only available to root. Try

sudo netstat -antlp

it should return the pid of every open port that's not in a TIME_WAIT state

or, if you want to know process ID related to specific port (let us say 8765 for example) use the code

netstat -tulpn | grep :8765

Solution 3

Based on hint from @user202173 and others I have been able to use the following to track down the process that owns a port even when it is listed as - in netstat.

Here was my starting situation. sudo netstat shows port with PID/Program of -. lsof -i shows nothing.

$ sudo netstat -ltpna | awk 'NR==2 || /:8785/'
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp6       0      0 :::8785                 :::*                    LISTEN      -
tcp6       1      0 ::1:8785                ::1:45518               CLOSE_WAIT  -
$ sudo lsof -i :8785
$

Now let's go fishing. First let's get the inode by adding -e to our netstat call.

$ sudo netstat -ltpnae | awk 'NR==2 || /:8785/'
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode       PID/Program name
tcp6       0      0 :::8785                 :::*                    LISTEN      199179     212698803   -
tcp6       1      0 ::1:8785                ::1:45518               CLOSE_WAIT  0          0           -

Next use lsof to get the process attached to that inode.

$ sudo lsof | awk 'NR==1 || /212698803/'
COMMAND      PID    TID                USER   FD      TYPE             DEVICE   SIZE/OFF       NODE NAME
envelope_ 145661 145766               drees   15u     IPv6          212698803        0t0        TCP *:8785 (LISTEN)

Now we know the process id so we can look at the process. And unfortunately it's a defunct process. And its PPID is 1 so we can't kill its parent either (see How can I kill a process whose parent is init?). In theory init might eventually clean it up, but I got tired of waiting and rebooted.

$ ps -lf -p 145661
F S UID         PID   PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD
0 Z drees    145661      1  2  80   0 -     0 exit   May01 ?        00:40:10 [envelope] <defunct>

Solution 4

I don't know what these are specifically, but kernel modules (NFS for example) do not have a PID to associate with these sockets. Look for something suspect in lsmod.

Solution 5

I dont know if this can be useful. I had the same problem and what I did is the following: First, I called netstat with options -a(all) and -e(extended). With the latter option I can see the Inode associated to the used port. Then, I called lsof |grep with the inode number obtained and I got the PID of process associated to that inode. That worked in my case.

View more solutions

Comments

[CrazyTux]

A given solution for the specific problem - serverfault.com/questions/1078483/…

[user3398902]

I did try to locally netcat to both ports and it displays nothing. For the tcp port, it closes if I type anything and then enter. The UDP one only closes if I press Ctrl+C. I have iptables in place, and it does not allow connections to those ports, so unless they are bypassing iptables, I can't imagine something is connecting to them.

[user3398902]

lsmod returns nothing. This server is an NFS client. That is currently my #1 suspect.

[user4250202]

That would explain why the client ports have changed after a new instance of the kernel.

[user3398902]

It is a web server running apache and pretty much nothing else other than things like cron and syslog.

[petrus]

every open TCP ports only with this command. UDP ports will not be shown.

[Peter Hansen]

You shouldn't have been downvoted as this is a totally legit answer. It helped me find a case where the other answers (using rpcbind or lsof) didn't help. (And yes, it was NFS.) Thanks!

[Ryan Walls]

If you have this problem and want to force nlockmgr to use specific ports, try this solution: fclose.com/39625/fixing-ports-used-by-nfs-server.

[SamB]

Hmm, I wonder why it doesn't assign a PID to the NFS client just so you can see what's up with that ... I guess that'd require it to have a worker thread or something?

[manish Prasad]

Have been searching for this solution for months now. thank you for this jem

[Stefan Lasiewski]

I've been looking for a solution like this for years. Thank you! One of the downsides here is that if an NFS client or server is bogged down, then lsof | awk 'NR==1 || /212698803/' (even with lsof -N to show NFS only) can be very slow to respond and may time out. Another downside is that the inode may change while you are troubleshooting this.