How to achieve multiple NFS/TCP connections to the same server?

Question

I'm trying to maximize the NFS/TCP bandwidth between a Linux client and a single NFS server over a 10Gbps link. Although each NFS mount has its own device (major/minor), everything comes down to a single TCP connection and this is a bottleneck.

Using iperf and multiple TCP connections, I can reach ~8Gbps, but NFS is stuck on a single TCP stream and limited to ~2Gbps (no tcp tuning yet).

On Solaris, there's the system setting rpcmod:clnt_max_conns which allow to set how many concurrent TCP connection the system is able to use to a single NFS server.

Is there a Linux equivalent or a way to do so ?

As a workaround, I know I could set many IP on the NFS server, and then load-balance the mountpoints on them but that wouldn't be optimal.

Solution 1

Starting from linux kernel 5.3 a new mount option "nconnect=X" is available, where X specifies the number of TCP connections to the server to use.

Solution 2

You could switch back to UDP and use jumbo packets (4k datagrams for instance). Having your whole network support jumbo frames might help a lot, so that you don't have IP packets reassembly on each side.

Comments

[Benoît]

Unfortunately, I have to stick with NFSv3.

[shodanshok]

Good suggestion. Ideally, jumbo frames should be at about 9K. This will maximize its network throughput.