Resolving high iowait on Samba server

Question

We've got a centralised file cache which a cluster of web servers use to store "heavyweight" pages. Each webserver uses Samba to mount this shared area.

We're getting a lot of iowait on the server, and I wondered what steps we could take to make a more efficient centralised cache? We're already using memcache as first line cache for some objects, and may simply throw more memory at that, but I'm interesting in finding out what techniques we could use to speed up a file based cache. All the servers run recent release of Ubuntu.

The server uses an ext3 filesystem with LVM. Maybe other filesystems would be more performant for this sort of activity? We used Samba for many years simply because everyone is comfortable with it and we had maintenance headaches with NFS (refusal to unmount, for example). Maybe there's better technologies...

Solution 1

Check the io scheduler (called the io elevator in the kernel documentation) assigned to that volume.

$ cat /sys/block/sda/queue/scheduler 
noop anticipatory [deadline] cfq 

For most RedHat and Fedora distros the default is the CFQ scheduler. IMHO this is not the best for an iobound server process. I recommend the deadline scheduler

$ echo "deadline" > /sys/block/sda/queue/scheduler

to change this, or add

elevator=deadline

to your boot arguments to make this permanent.

High iowait times can also be caused by aggressive readahead, which may be unnecessary for your workload. The default for RedHat and Fedora systems is 128k

$ cat /sys/block/sda/queue/read_ahead_kb 
128

Experiment with echo'ing lower values to that file and see if that decreases your iowait times.

Also, check the underlying disk or array. If it is degraded, or rebuilding, your iowait times will skyrocket as the underlying disk subsystem steals io bandwidth while rebuilding itself

Solution 2

A high IOWait% on the server means that you need faster disk / more memory for the disk cache.

Take a look at your 'iostat -x 5' statistics - are you saturating the disk? Depending on your application semantics, perhaps more memory on the file server will do you well.

If there's a long delay between storage & retrieval of pages by the server, then you need faster disk.

Check the output of free to see how much memory you have and what it's being used for.

Solution 3

Being that Samba is accessed over the network, I would consider that your network is the bottleneck first. Are the links optimized around the types and sizes of files/packets coming across the network?

Can you monitor the links to see how utilized they are?

If the network does not seem to be the bottleneck, than you can look at some of the tips in this guide to optimize your Samba configuration a little bit.

edit

You mention LVM, what type of disks are they? Regular off the shelf 7200RPM IDE drives? If it is truly IO wait, then yes, you would need faster disk. This could be as simple as a SATA 10k drive, or even going into a high performing raid.

Comments

[Steve Townsend]

By 'a lot of iowait on the server', do you mean the file server that is exporting the samba shares?

[Paul Dixon]

Yes - top typically shows something like this 9.2%us, 5.7%sy, 0.0%ni, 17.3%id, 66.8%wa

[Paul Dixon]

We see peaks of 80Mbps on the NIC, but looking at the server right now, we have 61% wait, but only around 20Mbps flowing through the NIC.