Can TCP provide more than 65535 ports?

Question

Is it possible to setup a Linux system so that it provides more than 65,535 ports? The intent would be to have more than 65k daemons listening on a given system.

Clearly there are ports being used so this is not possible for those reasons, so think of this as a theoretical exercise in trying to understand where TCP would be restrictive in doing something like this.

Solution 1

Looking at the RFC for TCP: RFC 793 - Transmission Control Protocol, the answer would seem to be no because of the fact that a TCP header is limited to 16-bits for the source/destination port field.

    

Does IPv6 improve things?

No. Even though IPv6 will give us a much larger IP address space, 32-bit vs. 128-bits, it makes no attempt to improve the TCP packet limitation of 16-bits for the port numbers. Interestingly the RFC for IPv6: Internet Protocol, Version 6 (IPv6) Specification, the IP field needed to be expanded.

When TCP runs over IPv6, the method used to compute the checksum is changed, as per RFC 2460:

Any transport or other upper-layer protocol that includes the addresses from the IP header in its checksum computation must be modified for use over IPv6, to include the 128-bit IPv6 addresses instead of 32-bit IPv4 addresses.

                 

So how can you get more ports?

One approach would be to stack additional IP addresses using more interfaces. If your system has multiple NICs this is easier, but even with just a single NIC, one can make use of virtual interfaces (aka. aliases) to allocate more IPs if needed.

NOTE: Using aliases have been supplanted by iproute2 which you can use to stack IP addresses on a single interface (i.e. eth0) instead.

Example

$ sudo ip link set eth0 up
$ sudo ip addr add 192.0.2.1/24 dev eth0
$ sudo ip addr add 192.0.2.2/24 dev eth0
$ ip addr show dev eth0
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc
      pfifo_fast state DOWN qlen 1000
    link/ether 00:d0:b7:2d:ce:cf brd ff:ff:ff:ff:ff:ff
    inet 192.0.2.1/24 brd 192.0.2.255 scope global eth1
    inet 192.0.2.2/24 scope global secondary eth1

Source: iproute2: Life after ifconfig

References

• OpenWrt Wiki » Documentation » Networking » Linux Network Interfaces
• Some useful command with iproute2
• Linux Advanced Routing & Traffic Control HOWTO
• Multiple default routes / public gateway IPs under Linux
• iproute2 cheat sheet - Daniil Baturin's website

Solution 2

Is it possible to setup a Linux system so that it provides more than 65,535 ports?

Nope.

The intent would be to have more than 65k daemons listening on a given system.

Then you need:

• an iptables configuration that redirects on traffic content or

• a "service broker service" or "multiplexor service" that will accept incoming connections on a single port and route it to the appropriate daemon "behind it". If you want standard protocols to pass unmodified you may have to implement protocol sniffing/recognization in this multiplexor service, in a fashion that an IDS or layer-7 firewall would anaylze; completely possible with the great majority of protocols.

Per the second item, you could design this service to handle more than 2^16 "ports" if you really wanted to. I'm sure the performance impact will be minimal compared to the load of 2^16+ listeners running.

Daemons in Linux can be listening on unix sockets which exist in the filesystem, so your "multiplexor service" could maintain an internal mapping of external port <-> internal unix socket. You'll likely run into a kernel process limit (32Kbyte processes?) before running out of inodes on any modern filesystem.

Solution 3

Just because there is no good answer I wanted to chime in.

One way to do this would be to add an IP option which specifies the port extension. The option must be designed to fit within the optional portion of the IP header and would be skipped by unknown hops.

You would use this option and it's information information to extend the source, destination or both port numbers.

The limitations are not going to automatically work in existing software just by adding the option anyway, they will have to be rewritten to take advantage of the option no matter how it's implemented, existing software and firewalls will either ignore the packet or process it as usual using the value in the source and destination port fields.

In short it is not easy to do and would be better done using a single reusable listener and data contained in the payload of the packet.

You can also more easily allow port reuse in the software, which can help to overcome this limitation by reusing ports of the server for multiple client connections.

Rtsp for example can use the SessionId header in conjunction with various other headers in the payload of the IP packet to determine what connection the request was issued for and act accordingly e.g. if the socket from which the message was delivered is not the same as the socket's remote address to which the session corresponds then then one can either allow a session to be updated with the new socket for processing, deny the message or a variety of other actions depending on the application.

An Http server can also do this or any other type of server.

The key thing to remember when allowing reuse of ports is that you must also take into account the source IP address.

Comments

[Admin]

What is the motivation for this question? Why do you want to have that many daemons listening?

[Admin]

Also, you're going to have a hard time starting that many processes. (I assume you mean one process per daemon.)

[Admin]

While there is nothing that formally restricts you from wearing 65 pairs of trousers at once, it would be practical idiocy to try. If you can show me a machine that can fruitfully process 10'000 TCP ports concurrently, then this might be an interesting abstract question.

[Admin]

The nature of this Q is completely theoretical, no intended purpose other than to understand the limitations of TCP & the # of ports.

[Admin]

The thing is, though, you have phrased it in a way that ties it to various practical matters involving RAM space required by 64k+ daemon processes. Any machine you're likely to have now or for the next decade or so will run out of RAM before you hit the listener limit. If you rephrase the question to talk only about TCP listeners, leaving the talk about daemons out of it entirely, that problem goes away. You can amortize stack space by assigning a thousand sockets to each single-threaded event-driven daemon, for instance.

[Admin]

@WarrenYoung - that's doubtful 8-). I have access to many systems with 256GB RAM or more if needed. Truth this Q is a proxy for the A that I wanted to write. This Q came up in the chatroom and while researching it I found little that explained why so I was creating that content on the interwebs. Even systems with 48GB or 64GB are readily available, we have several at my day job.

[Admin]

I know one way to find out. :)

[Admin]

@WarrenYoung - yeah we can always try can't we 8-)

[Admin]

@msw WhatsApp handle 2 million TCP connections on one machine, as described here: stackoverflow.com/questions/22090229/…. I know ports != connections, but it makes no difference to the question of whether this is practical.

[Admin]

@Bryan that's a different issue. That is many connections to a single listening port, this question is many listening ports.

[Admin]

@slm if the goal is purely to have >64k listening daemons, there is SO_REUSEPORT. You can have multiple daemons listen on a single port. Therefore your limit is number of processes. You can also fork after listening and accomplish the same thing.

[Admin]

Do you mean > 64K daemons in the sense of that many distinct services, or multiple processes on the same service?

[Admin]

@RussellBorogove I mean I have 64k+ daemons listening and that they would have simultaneously 64k+ connections to these ports all at the same time.

[Admin]

@Patrick - does SO_REUSEPORT mean that only one at a time can use the port though?

[Admin]

@slm SO_REUSEPORT means multiple programs can listen/accept on the port at the same time. The kernel will round-robin through them each time a connection comes in. You get the same behavior if you open the socket and then fork.

[Admin]

@Bryan I can't help but think the WhatsApp example you cite is akin to me claiming "I can encode real numbers up to about 1.8 * 10^308 in a 64-bit IEEE754 double". That statement isn't strictly false, but what is unstated is at least as important as that which is.

[Admin]

@bryan Actually, it makes all the difference, and you don't seem to have read either that question or its answers very well. When WA claims to handle 2 million TCP connections (with 'simultaneously' strongly implied but left unsaid) what they mean is they've tweaked their server software and processing power to optimize response times so that none of said two million connections will time out before the last one is served. They're almost certainly all connecting to one single port on that server, though.

[Admin]

in theory once you run out, you can start virtual machines with a new ip address and listen on these machines.

[Admin]

Patrick/Shadur/@msw Are asking why Bryan quoted stackoverflow.com/questions/22090229/…. @msw has a comment above with 13 upvotes claiming that 10,000 concurrent connections is not feasible easily. You people must be living in 1980. Here's another link so you can update to the year 2019 stackoverflow.com/questions/1575453/…

[Admin]

It would be an interesting separate question to ask why 65k server-side ports would be ever necessary. A single IPv4:Port server TCP listener is sufficient: According to IPv4 and TCP network packet fields, a single IP:Port can can serve in the order of 281 trillion connections (2^32 * 2^16) concurrently.

[Admin]

One great example of using multiple server-side port numbers, is NAT Port Forwarding: you might have each public IP TCP port forwarding to a different internal remote desktop service. In fact, in Australia Cellular ISPs use NAT'ing heavily to drive their public IP pool further, however, here they are limited by a total number of client-ports going out to the internet. IPv6 helps for both port-forwarding and cellular ISPs. One can economically obtain a subnet of IPv6 addresses, binding each to internal hosts.

[supercat]

It would not be possible to select among 65,536+ daemons using the destination port alone, but if one had unlimited memory and bandwidth, could have over 32,000 connections with every distinct TCP address on every incoming port.

[user3922906]

I downvoted this because you say it's not possible, then go on to explain how to do it using multiple IPs and load balancing, albeit in a very confusing roundabout way.

[LawrenceC]

More than 64K ports on a single system is impossible. More than 64K listeners is probably possible, but you have to have proxy or frontend listeners that would "split" incoming connections to the right real "backend" listeners. You could do something insane like an internal NAT to multiple internal IP addresses, for example.

[user3922906]

Wrong. People have managed to get half a million concurrent connections on a single system. Yes, multiple IPs and load balancers (not necessarily on the same system) are required, but a single system can open more than 64k ports and even more than 64k listeners if done right.