iostat shows 100% utilization - what can be done to improve performance?

Question

I have dedicated web server with CentOS.

Recently, I get very high server load, probably caused due to IO bottleneck. iostat -mx 3 shows:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          12.15    0.00   19.97   67.89    0.00    0.00

Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sda              15.33     6.67 32.67  0.67     2.58     0.03   160.00     0.27    8.20   7.03  23.43
sda1             15.33     6.67 32.67  0.67     2.58     0.03   160.00     0.27    8.20   7.03  23.43
sdb             134.67     0.00 145.33 10.67    13.34     0.19   177.59   131.25  963.31   6.41 100.03
sdb1            134.67     0.00 145.33 10.67    13.34     0.19   177.59   131.25  963.31   6.41 100.03
sdc               0.00    17.33  0.33 21.00     0.00     0.15    14.50     0.08    3.84   0.31   0.67
sdc1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdc2              0.00    14.33  0.33  9.33     0.00     0.09    19.86     0.04    4.28   0.48   0.47
sdc3              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdc4              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdc5              0.00     0.67  0.00  4.33     0.00     0.02     9.23     0.02    4.85   0.46   0.20
sdc6              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdc7              0.00     2.33  0.00  7.33     0.00     0.04    10.55     0.02    2.68   0.18   0.13
sdc8              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

what can be done to improve performance? My web server reads lots of big files from that HD drive. does 13MB/s for HDD sata2 is a reasonable value? I would expect it to be higher.

top output (head only):

Cpu(s):  7.5%us,  5.5%sy,  0.0%ni,  4.1%id, 71.5%wa,  3.0%hi,  8.5%si,  0.0%st
Mem:   2048144k total,  2036732k used,    11412k free,    23732k buffers
Swap:  8385888k total,      192k used,  8385696k free,  1375064k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
15369 apache    15   0  116m 9.8m 3564 S  3.3  0.5   0:01.37 httpd
23491 apache    15   0  117m  10m 3644 S  3.3  0.5   0:00.55 httpd
21694 mysql     25   0  308m  49m 4752 S  3.0  2.5   1068:49 mysqld
25735 apache    15   0  117m  10m 3660 S  2.7  0.5   0:00.53 httpd
25909 apache    15   0  116m 9964 3548 S  1.0  0.5   0:00.15 httpd
14814 apache    15   0  117m  10m 3684 S  0.7  0.5   0:00.53 httpd
15708 apache    15   0  117m 9.9m 3564 S  0.7  0.5   0:00.66 httpd
21464 apache    15   0  117m  10m 3664 S  0.7  0.5   0:01.32 httpd
22997 apache    15   0  116m 9.8m 3656 R  0.7  0.5   0:04.16 httpd
23484 apache    15   0  117m  10m 3668 S  0.7  0.5   0:00.28 httpd

thanks

Solution 1

What iostat is basically showing here is your disk saturated with IOPS.

man iostat:

• rrqm/s — The number of read requests merged per second that were queued to the device.

• r/s — The number (after merges) of read requests completed per second for the device.

HDDs IOPS performance reference data, Wikipedia

Compare the numbers.

Solution 2

I know its a pretty old thread but I am facing similar issue

Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util

sdb             134.67     0.00 145.33 10.67    13.34     0.19   177.59   131.25  963.31   6.41 100.03

So in my case whenever avgqu-sz reaches equal to lun queue depth(/sys/block/sdb/device/queue_depth),disk saturation take place and in my case it's set to 128.I just increase it to 192 and now getting better performance.I know it's not a better solution but I can't afford to change disk currently so this is the only solution I have.

Comments

[mdpc]

Are you swapping? You might want to include a snapshot of top.

[Niros]

top output: top - 21:02:52 up 60 days, 15:43, 2 users, load average: 78.72, 72.56, 70.79 Tasks: 313 total, 3 running, 310 sleeping, 0 stopped, 0 zombie Cpu(s): 9.8%us, 6.8%sy, 0.0%ni, 0.0%id, 70.8%wa, 2.7%hi, 10.0%si, 0.0%st Mem: 2048144k total, 2034540k used, 13604k free, 14428k buffers Swap: 8385888k total, 192k used, 8385696k free, 1430824k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 21694 mysql 25 0 308m 49m 4752 S 5.0 2.5 1067:53 mysqld 6897 apache 15 0 117m 9.8m 3528 S 4.6 0.5 0:00.38 httpd

[JustMyThoughts -]

@Niros, please update the question instead so we can benefit from the correct formatting. Cheers

[tink]

If your Centos install has iotop available I'd have a look at its output, too. Also, is there anything in dmesg' output regarding the HDD, and its performance? Bus resets?

[Michael Hampton]

Exactly what hard drives are you using, and with what controller?

[Niros]

HD model: WD2002FAEX, I don't know what controller is (RAID controlloer? I don't have one) I couldn't find iotop command in my centos (5.8) dmesg full with Firewall block messages.

[Niros]

I have the follwoing HDD: wdc.com/global/products/specs/?driveID=899&language=1 I don't have raid controller. What should I do in order to achive 40 MB/s? with hdparm -t I get: Timing buffered disk reads: 272 MB in 3.00 seconds = 90.63 MB/sec

[JustMyThoughts -]

bonnie++ is a much better benchmark of IO. Nonetheless, I think you need to spread your load across spindles to help your random access times. This can only be achieved with RAID

[Niros]

What about defrag? maybe the files are fragmented? In addition, is there anything that I can do to improve performance with my current hardware? changing block size might help?

[JustMyThoughts -]

Fragmentation is very unlikely and is not something that concerns most Linux engineers. Judging by the amount of httpd servers you've got running, you can't avoid all those clients trying to access different parts of the filesystem. If you don't want to spend the money then perhaps you should reduce the number of clients.

[Niros]

what about converting disk from ext3 to ext4? what about mounting the disk with noatime? what about changing blcok size? thx

[JustMyThoughts -]

why don't YOU try it?

[wazoox]

That's a perfectly correct solution to enlarge the queue, if your disk is capable of managing more events in queue.