How to close a port with no process attached?

Question

This question is similar to Network port open, but no process attached? and netstat shows a listening port with no pid but lsof does not

I tried all i can do(as root: netstat, lsof, ls -al /proc/*/fd etc.), but i can't find the pid.

Anyway, i have to close or release the port without pid, because my process want to listen on it. Is anyway to do it?

I don't want to reboot the server. Because there is a process in my system, it will update bin files in all of my servers then deploy all servers automatically. The deployment will be failed when 7123 or some other ports were listened.

Thanks.

More Details

There are three servers in this situation, all of them are Rackspace's servers, and all 'bad port' is udp port 7123.

Reboot can solve this problem, i have tried it on one of those three servers. But i don't want to reboot the server.

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
udp   213120      0 134.213.205.214:7123        0.0.0.0:*                           -

The result of nmap shows this port is not closed.(@Enzo)

root@auto:~# nmap 134.213.205.214 -p 7123 -sU

Starting Nmap 6.40 ( http://nmap.org ) at 2018-04-16 12:01 UTC
Nmap scan report for auto (134.213.205.214)
Host is up.
PORT     STATE         SERVICE
7123/udp open|filtered unknown

Nmap done: 1 IP address (1 host up) scanned in 2.12 seconds

Update

For now, reboot is the only way to close/release this port. But reboot is a bad idea, it makes service stop.

Solution

There's the possibility your system is running in an inconsistent status. I would start by doing some basic diagnostics.

Check the UDP socket and its process.

Run sudo netstat -lunp or sudo ss -lunp to see whether that UDP socket (on port 7123) is busy. sudo is needed for a normal user to escalate privileges and see the PID and the process name who's "listening" on that port. Without sudo there will be a - instead of the PID and process name. Take note of the the PID. In case there's none listed, then you can start thinking your system has been compromised as the process is capable of hiding itslef by manipulating the internal kernel structures for the processes!
If you cannot see the PID with high privileges, than you'd better to insulate that server as it's probably been compromised and hacked.

Check whether the listening process is also "talking"

I suggest to use nmap or the more general netcat tool. With nmap you can try this command:

sudo nmap -sV -v -Pn -sU -p U:7123 134.213.205.214

It tries to understand the protocol spoken at port UDP:7123. sudo is required for the specific scan type (UDP service scan). If you see data that don't match what you are expecting, than probably either the process is gone wild or is in an inconsistent status. Or it's been replaced by some other (malicious?) code.

With netcat you need some manual intervention to generate traffic to the server:

netcat -u 134.213.205.214 7123

Whatever you'll type will be sent to your application on port UDP:7123. You can also pipe some random data (but don't expect any meaningful result) with:

cat /dev/urandom | netcat -u 134.213.205.214 7123

Check the system and application logs (if any)

The former are usually stored at /var/log/ directory. Application log could be anywhere else. The tool lsof can be of help if you check the PID on its 2nd column. I'd keep an eye on these logs during the subsequent actions.

Check the process table for that PID

The command ps can provide for a lot of details about processes. My personal favorite is:

PS_FORMAT="ruser,pid,ppid,s,%cpu,rss,cmd" ps ax --sort=pid

You can see on the second column the PID you are looking for, along with the real user ID (ruser, 1st column), the parent PID (ppid, 3rd column), the status (s, 4th), the % CPU usage (%cpu, 5th), the resident set size (rss, 6th) and the command line with arguments (cmd, 7th). In my opinion, for this very case, the process status (it's a single letter) and the percentage of used CPU are the key values, along with the command line.

Check on the man page for all the details for ps and to fine tune the output.

Check the binaries

If you have another machine with the same architecture and OS and the same process running in an expected way, you can check whether the binaries match byte-by-byte. In case they don't, you'd better reinstall those binaries from a known secure source.

Let's assume the program is /usr/local/bin/myserver.
Calculate and take note of its checksum. Something like this:

sha512sum /usr/local/bin/myserver

If the program has been compiled to use dynamic libraries, then you'd also check them. The list of used dynamic libraries is obtained with:

ldd /usr/local/bin/myserver

Be warned: the output can be quite lengthy, but for each line you'd calculate and take note of the checksum for comparison.

In case any discrepancy with the reference trusted system is found, I'd suggest for a re-installation of the whole system and applications. It's a radical approach but I think that the target system cannot be trusted any more.

Kill and restart the process

I would try to kill the rogue process (provided that you know its PID) and restart it to check whether the rogue behavior sticks. In order to kill a process given its PID you can run:

sudo kill -s SIGKILL <PID>

I need to warn you that there is a number of cases in which a process won't die or will die later than expected. It depends mostly upon the process status shown by the ps command run earlier. The referenced man page reports some information about the process status column at the "PROCESS STATE CODES" paragraph.
Use the command ps to check again the process status. It should normally be kicked out within a few seconds and disappear from the process list.

Once the process has been killed you can try to restart it and see whether the rogue behavior repeats.

As a general rule of thumb, if you don't change anything to the system, the behavior will hardly change as well. That is, just rebooting the application or the system will hardly solve the problem: it will just be pushed forward in time.

Comments

[Admin]

Have you tried to use nmap over that port to see what happens? If there's no answer at all, then it's actually closed, unless a listening process will pop up and listen again. In this very case, to close the port, you can use iptables to deny/drop all of that traffic.

[Admin]

@Enzo Thanks for reply. I put the result of nmap in description. In this case, my process want to listen this port, so i can't use iptables to drop packets. I have to release this port.

[Admin]

Are you asking how to write an iptables rule, or how to find the process holding the socket open? It will be visible in lsof -n | grep 7123 unless you have a rootkit installed or unless that is an orphaned file descriptor which should be easy to spot with ls -al /proc/*/fd, both commands being run as root of course.

[Admin]

I'd start closing (DENY) that port with iptables. Just to be sure the service is unavailable. Then, can re-run nmap with "-A -v" to get more details?

[Admin]

@Aaron Thank you for reply. No, i'm not asking how to use iptables because in this case, my process want to listen at 7123. So iptables will not help for that. I've tried lsof and netstat` run as root. If ls -al /proc/*/fd's result not contain inode of 7123, is that means orphaned file descriptor? How can i release this orphaned file descriptor?

[Admin]

@degas, either you have a process capable of hiding itself from root (sudo lsof|grep -i udp) or your process is going zombie. I hope you're in the second case.

[Admin]

@Enzo After drop all incoming traffic to 7123 by using iptables, the result of nmap is no changes.I've tried find the process as root. And there is no zombie. Maybe reboot is the only way to recover this problem?

[Admin]

@Degas I meant, can you run your application so it uses a different port number and taking note of its PID? If neither ps nor netstat can show the process, then your system is likely to be FUBAR!

[Admin]

@Enzo I have 3000 more other servers run the same service and use the same port, they works fine. Emmm, i think you are right, this server is FUBAR...

[Admin]

@Degas, I have added an answer.