Bad disk performance on HP DL360 with Smart Array P400i RAID controller

Question

I have a HP DL360 server with 4x 146GB SAS disks and a Smart Array P400i RAID controller with 256MB cache. The disks are in RAID 5 (3 disks + 1 hot spare).

The server is running VMware ESX 3i.

The disk write performance is really bad. Here are some numbers:

ns1:~# hdparm -tT /dev/sda

/dev/sda:
Timing cached reads: 3364 MB in 2.00 seconds = 1685.69 MB/sec
Timing buffered disk reads: 18 MB in 3.79 seconds = 4.75 MB/sec

ns1:~# time sh -c "dd if=/dev/zero of=ddfile bs=8k count=125000 && sync"
125000+0 records in
125000+0 records out
1024000000 bytes (1.0 GB) copied, 282.307 s, 3.6 MB/s

real 4m52.003s
user 0m2.160s
sys 3m10.796s

Compared to another server those number are terrible: Dell R200, 2x 500GB SATA disks, PERC raid controller (disks are mirrored).

web4:~# hdparm -tT /dev/sda

/dev/sda:
Timing cached reads: 6584 MB in 2.00 seconds = 3297.79 MB/sec
Timing buffered disk reads: 316 MB in 3.02 seconds = 104.79 MB/sec

web4:~# time sh -c "dd if=/dev/zero of=ddfile bs=8k count=125000 && sync"
125000+0 records in
125000+0 records out
1024000000 bytes (1.0 GB) copied, 35.2919 s, 29.0 MB/s

real 0m36.570s
user 0m0.476s
sys 0m32.298s

The server isn't very loaded and the VMware Infrastructure Client performance monitor is showing 550KBps average read and 1208KBps average write for the last 30 minutes (highest write rate: 6.6MBps).

This has been a problem from the start. Any ideas?

Solution 1

My performance varies between 80 and 350MB/s for buffered read, you can go into the smart array tool on smart start and enable the various options, including cache with/without battery if you want for each logical drive, and on disk cache for the array which is normally off, you can do this after creating the array.

Solution 2

The write performance on that particular controller is usually poor unless you also have the battery unit for the cache. In addition, reconfiguring the array as a RAID 1+0 would give you the same amount of space and better overall performance.

Are you testing this from the ESXi console or from within a VM?

Solution 3

You don't have the optional BBWC do you? My performance under ESX4 was horrible (5mb/s reads/writes) without one. Its stellar with one (10x improvement). Without the BBWC the array controller disables write-back cache, and performance nosedives as a result.

As a heads up -- my server was also missing the $50 cable that goes from the BBWC to the P400. Make sure yours has one, if not, its cheaper to buy them together.

Comments

[ewwhite]

With 4 disks, I'd always go with RAID 1+0. I haven't experienced this with RAID5, but that will make a difference. In addition, I've seen all sorts of weird performance behavior when running these systems without the battery-backed cache. It's a must-have in most cases.