Latency on appearance of files written via NFS share

Question

I have two CentOS OS servers, one acts a NFS file server, and the other as development server. The NFS server and client are running v3 of NFS. When the client writes to a NFS share I'm seeing an odd latency issue where the the client completes it's write of a file, but on the server side I may not see the file appear for sometimes up to a minute afterwards. The time it takes for the file(s) to appear is somewhat random, but there is always a latency period between when the client writes and when it appears on the server. These files aren't of significant size, sometimes as small as a few Kb in size. The two servers exist on a 10Gbit switch and are configured with dual bonded 10Gbit connections for redundancy.

I'm at a loss.

Solution

There is an element of NFS caching in action. This is done to improve performance. If you want immediate update of data, change the mount options to add tell NFS to run in sync.

Also remember that if the program is using buffered I/O ... unless it does syncs and/or flushes its buffers, the time to update a local program cache may be variable. This is also done to improve performance.

Comments

[MadHatter]

Are all server and client clocks synchronised?

[Admin]

To avoid confusion: When you say 'client' you mean one of the two CentOS servers, the one mounting the remote NFS share and when you say 'server side' you mean the other CentOS server that is exporting the NFS mount?

[Michael Martinez]

The first thing I'd do is make sure your forward and reverse DNs lookups are configured for both hosts. A lot of times when one or the other isn't working, you'll see weird latency issues like this (I've seen it frequently in ssh connections.) ... Second, I'd put a packet sniffer on both ends and figure out how long it takes the file to get on the wire (network) after being written out to host A.

[user218983]

Yes, client side is the server mounting the shares. Server side is the one exporting the NFS share.

[user218983]

These are great points. I am already exporting these file systems with sync defined. And the duration to which files appear across the network are random for all applications including OS commands like touch. One thing additional to add, these issues didn't exist when both hosts were running Cent 5.5. Currently the client, the computer who's mounted the file system is running Cent 6.5.