How can I improve NFS performance on a gigabit LAN?

Question

I have a Solaris 10 server with a Mac OS X clients on a gigabit LAN. The Solaris server exports a RAID-Z backed NFS share and all the clients connect to it. However, transferring files is slower than I think it should be. I enabled jumbo frames and that increased my maximum throughput from 18MB/s to 30MB/s but that's still slower than I think it should be. I think the current bottleneck is my switch, which doesn't seem to be able to switch as many packets per second as my machines can send. Also, both machines use about 30 - 40% CPU for the transfers.

Other than replacing the switch, which I plan to do, what can I do to decrease CPU usage and increase throughput?

Solution 1

A couple of things:

• To figure out whether the switch is the problem, take the switch out of the equation and test. Connect the machines back to back and test again. GigE cards will auto cross connect for you, no special cable needed.

• After that, NFS tuning is a bit of a black art. From what I have seen, it tends to be very specific to the NFS server and sometimes the NFS client. Luckily, Google seems to have many papers on the subject. Read a bunch.

Solution 2

Often the ZFS Intent Log (ZIL) is the NFS bottleneck of a ZFS file server. ZFS obeys the NFS protocol very strictly and makes sure that the data of synchronous writes really hits the disk before it reports success to the writer. ZFS uses the ZIL to temporarily store this data before it writes it in a bigger chunk to the pool. If writes to your ZIL are slow (high latency) then NFS performance probably will be slow, too. A good test case of this problem is to extract a large tarball (many files) to the NFS export.

There are at least two solutions to improve the NFS performance:

• Use a dedicated, fast storage device for the ZIL (think NVRAM or SSD). You can even create a ZIL mirror out of two devices. Warning: AFAIR it's not possible to remove a ZIL device after you've added it to a pool. Think twice. If you want to try SSDs for you ZIL you may want to read this interesting blog post: Somewhat stable Solid State
• Disable the ZIL on your ZFS file server. Well, this of course breaks the protocol promise of synchronous writes at the risk of data integrity but it'll be much faster. Read this part of the ZFS Evil Tuning Guide before your try this alternative and make sure you understand the consequences.

Solution 3

A couple of hints:

• try mounting the server via SMB/CIFS and seeing what kind of performance you get
• try doing a test using ttcp or iperf to see what your network bandwidth is like from client to server

OS X NFS is not fantastic in our experience (OS X clients, Linux servers). We get better performance by using samba. YMMV!

Comments

[Amok]

Thanks for the advice. I installed netatalk and that has improved the performance and is easier to work with.

[Amok]

After doing this I found out that the switch wasn't a problem. I did try a little NFS tuning, but since there doesn't seem to be much information about it I gave up and switched to netatalk instead.

[kbyrd]

Googling NFS tuning shows a bunch of hints, but it's usually specific to a particular NFS server or client.

[Amok]

I've had the opposite problem when I'm writing to the array. It seems that ZFS caches about 1.2GB of data before it writes to disk. While it's writing to disk my network traffic drops to almost zero.