Very Slow Virtual Disk I/O, LSI_SCSI Warning on P2V W2K3 64-Bit VMWare Guest

Question

I did a P2V conversion of a production server (W2K3R2 64-bit) a while back using the VMWare conversion tool. I then installed the free VMWare Server 2.0 product on a Windows server, and am able to boot and interact with the VM guest. My purpose for this project is to do testing of some patches to our ERP software.

Working with it a bit more, I'm seeing terrible disk I/O performance on the guest OS to the point of it being unusable. While the virtual disks are on a big fat 1TB SATA RAID1 mirror, I'm getting times of like two hours to copy 250MB files to the virtual disk while the same copy in the host OS environment only takes a minute or two. I don't expect blazing performance out of this test environment compared to the production system, so I'm okay with some degradation, but this is ridiculous.

I've tried the following:

1. Updated VMWare Server to 2.01.
2. Updated Virtual Hardware on VM to V7.
3. Updated LSI_SCSI driver to latest 64-bit version from LSI web site.
4. Reducing virtual CPUs to 1.

I still get this error when I Boot the VM:

Event Type: Warning
Event Source:   Lsi_scsi
Event Category: None
Event ID:   26
Date:       8/6/2009
Time:       11:01:46 AM
User:       N/A
Computer:   VULCAN-TEST
Description:
The driver has detected that device \Device\RaidPort0 has old or out-of-date firmware. Reduced performance may result.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 0f 00 18 00 01 00 6a 00   ......j.
0008: 00 00 00 00 1a 00 04 80   .......
0010: 00 00 00 00 00 00 00 00   ........
0018: 00 00 00 00 00 00 00 00   ........
0020: 00 00 00 00 00 00 00 00   ........
0028: 00 00 00 00 00 00 00 00   ........
0030: 00 00 00 00 1a 00 04 80   .......
0038: 00 00 00 00 00 00 00 00   ........

My assumption is that the terrible I/O performance is related to this complaint from the LSI_SCSI driver.

The host system is running Windows Server 2008 x64 on an Intel E5405 quad-core dual-socket box with one CPU.

I see lots of discussion of I/O issues with VMWare and x64 Windows Server guests but I've tried everything I've seen suggested and so far no joy.

Solution

You may want to install ESXi on the host server instead of fighting VMware Server. I have installed VMware Server 2.0 for 4 different clients on Windows 2003 and Ubuntu 8.10 host OSes. The hardware was simliar to to yours in that we were using 7200RPM SATA drives on a PERC RAID controller. When we tried running SBS 2003 and Windows 2003 guests on it the IO performance was atrocious (in the sub 10MB/sec category). When we installed ESXi and migrated the guest OSes on it IO performance jumped to 90MB/sec and the results were much better.

I don't know how flexible you can be with the OS on that server but you may end up saving much more time by just moving to ESXi.

Comments

[Brian Spolarich]

VMWare Tools are up to date on the guest. Actually I was incorrect - the VM disk files live on a RAID mirror (2x1TB SATA drives). File copies on that volume outside of the VM are pretty fast.

[Brian Spolarich]

Disk queues stay < 1 while the 250MB copy from the host to the guest takes an estimated 2 hours. Virtual Server doesn't give an obvious option to change the SCSI virtual hardware. The LSI_SCSI error only appears in the guest's event log since the host has an Areca ARC-1680 SATA/SAS storage controller. Option #4 would be a lot of work as I don't have any 64-bit hardware I could just throw at this on a whim.

[Zypher]

The disk may be a red herring. Can you try attaching a second VHD to the machine and do a copy from one to the other see if you still have bad performance?

[Brian Spolarich]

I redid the P2V conversion. Tried an IDE disk instead of SCSI. Same badness. This makes me sad.

[Chopper3]

try a fresh, small, install of the same OS into a new VM on the same host/disks - how does that behave?