NFS write performance

Question

We have three VMware ESXi 4 hosts serving VM's from an OpenFiler NFS share. Every host has a direct gigabit connection to the NAS. Although read performance has been great, writing files inside the VM guests are suffering.

The recommended configuration for data integrity is to export NFS shares with the sync option and mount ext3 with data=journal.

I'd like to compare the behaviour of the maximum integrity configuration with the maximum I/O performance configuration. To configure for performance I exported the NFS share as

/mnt/raided/main/vm 10.0.0.0/255.255.0.0(rw,anonuid=96,anongid=96,secure,root_squash,wdelay,async)

while ext3 is mounted with

/dev/raided/main /mnt/raided/main ext3 defaults,usrquota,grpquota,acl,user_xattr,data=writeback,noatime

Will these configuration options give me optimal I/O performance? How about changing the file system? Will XFS improve performance significantly?

Other than the NAS crashing or power failures, what can cause data integrity issues with this configuration?

Solution 1

My understanding is that OpenFiler is based on the CentOS/RedHat family of OS builds. Linux's nfs server isn't the best to begin with*, and CentOS/RedHat's is worse than average.

*= when compared to NetApp filers, *bsd, or Solaris servers.

(Wonders if I'm going to get down-modded for that.)

Solution 2

I would think hard about using sync. I'm reminded about the Dirty Harry 'Do you feel lucky' speech.

NFS v2 and v3 is designed such that when the write is acked from the server to the client the data is on disk. This allows NFS to be stateless, and, therefore, the server COULD reboot between each request. One hopes not, but it could happen.

This means that if the client sees the ACK on the write then it doesn't have to care anymore about the data being on disk.

If you use async then this is not true anymore. It will be way faster though. Basically though if you use async, and, the server crashes, you probably should reboot the clients unless you know exactly what they do since if they are expecting to keep multiple files in sync the client may believe they are in sync, when, in fact, they are not.

Solution 3

yup, 'sync' is definitely a huge write-performance killer. I wouldn't ever consider it unless the data integrity demand is high. If you don't have battery-backed RAID and UPS, don't even bother because it will have little or no effect anyway.

'ext3 data=journal' doesn't help neither. If you crave for write performance, ditch ext3 and go for xfs instead, it's so much faster it's not even funny.

Comments

[Jan Jungnickel]

You fail to mention the level of redundancy you are using to provide the storage. RAID-5 has poor write characteristics.

[Hans Malherbe]

I'm using software RAID 10

[Arie K]

Our Openfiler+ESXi crude test showed ~30% improvement using XFS than ext3. Nevertheless, we decided to use iSCSI target feature. YMMV.

[Hans Malherbe]

I'd like to know how you tested as I don't have good ones.

[Arie K]

We used dd command from Linux VMs to write a large file. Hence the 'crude'.

[Hans Malherbe]

I'm a big fan of data integrity. The above mentioned performance exercise gives me an idea of what the cost is. I am quite shocked actually. I'm considering striking a balance by splitting the important VM's from the easily replaceable ones.

[Hans Malherbe]

Have you got a reference for the *bsd comparison?

[David Mackintosh]

Sorry, no documentation, it just something my *bsd-using cohorts happily claimed while I was fighting with a linux nfs server. No one has claimed any different as far as I am aware.