Analyzing Linux NFS server performance

Question

I'd like to do some analysis of our NFS server to help track down potential bottlenecks in our applications. The server is running SUSE Enterprise Linux 10.

The kind of things I'm looking to know are:

• Which files are being accessed by which clients
• Read/write throughput on a per-client basis
• Overhead imposed by other RPC calls
• Time spent waiting on other NFS requests, or disk I/O, to service a client

I already know about the statistics available in /proc/net/rpc/nfsd and in fact I wrote a blog post describing them in depth. What I'm looking for is a way to dig deeper and help understand what factors are contributing to the performance seen by a particular client. I want to analyze the role the NFS server plays in the performance of an application on our cluster so that I can think of ways to best optimize it.

Solution 1

Just an idea, try to sniff nfs traffic with wireshark. Might tell you which user accessed what file:

tshark -R nfs -i eth0

Solution 2

I have to say of all the different *stat utilities available to one, nfsstat is by far the worst! It gives you the ability to look at a bunch of counters but that's all. If you look at them twice, YOU have to do the work of trying to figure out by how much each counter changed and if you want to know the rate of change you then need to divide by the number of seconds between samples. In all fairness, nfsstat does date back many years when things were still pretty crude and is now hampered by nobody wanting to change the output format because it would probably break a lot of things.

As for using collectl to monitor nfs, it does provide nfsstat output in a much easier to read format, but what's even better you can let it run for hours or days and play back the data you had collected in the background. As for the request to see what processes are doing, collectl can also gather process data including how much I/O each process is doing and even play it back showing the top I/O users. You can also use the top feature in real time.

If you want to watch the disks theme selves collectl can do that too and display everything in a coordinated display.

Check it out... -mark

Solution 3

collectl (especially its NFS subsystem) is a very nice utility which might be useful for your analysis but it does not match your requirements list. I'm not aware of any Linux utility that does.

(Please let me add this off-topic note: There is software which matches your requirements: Sun's DTrace-based Analytics (pdf) - but unfortunately is not available on Linux. You'll find lots of great examples in Brendan Gregg's blog which illustrate the capabilities of this tool.)

Solution 4

In my opinion this exactly highlights the problem with today's tools. Here we're mentioned at least 3 including nfsstat, iostat and iotop. Then there was passing mention of wireshare and nfsreplay. Does this really sound like a normal way to do things? Other than wireshark with is a category all its own, wouldn't you prefer 1 tool?

For openers, while I find the output of iostat very useful, it's too hard to read with all those .00 in the numbers. Collectl reports the exact same data but formatted much easier on the eyes. You already know what I think of nfsstat and since collectl can play back any data there's no need for a 'replay' utility. As for 'iotop', collect can also show processes sorted by anything included I/O.

So there you have it all in too, complete with timestamps. If you need a finer monitoring interval you can alway crank back the sampling to 0.1 or 0.5 seconds or anything in between, though you will generate more overhead if you monitor processes this fast, but would with any process monitoring utility.

AND the final bonus is anything you collect with collectl you can load into a spreadsheet and easily plot OR use colplot which is part of collectl-utils.

-mark

Solution 5

You might want to try nfswatch from http://nfswatch.sourceforge.net

You can see some sample output at http://prefetch.net/blog/index.php/2009/06/16/monitoring-nfs-operations-with-nfswatch/

nfswatch is kind of like top (though I'm not sure if there's a batch mode). Once it's running you can change change the display by hitting a key (e.g. "c" to display NFS clients using your NFS server).

In my brief testing, however, nfswatch doesn't seem to work with NFSv4.

View more solutions

Comments

[Cian]

This seems like the kind of thing that system tap was written for. The docs are a bit crap, but I'd guess you could whip something up to do this kind of analysis using it. sourceware.org/systemtap/examples/keyword-index.html

[Læti]

Interesting suggestion. Actually, the author itself says NFSv4 is not supported and the tool hasn't been updated for about 3 years. A pity as it would be of great use!