Performance issue of Server 2012 VM on ESXi 5.1 U1

Question

I'm having issues with a VM, as stated in the topic, that runs on ESXi U1, build 1065491. The machine is sluggish and i can't figure out why.. Here's the details.

It runs on a Dell T420, has 12X1.9GHZ cpu's (it's a E5-2420). License is vsphere essentials, so i'm limited to 8vpu per host. I got a few vm's, but the cpu is not choked on the box it runs pretty smooth. All of the other vm's are snappy.

I allocated 40gb of ram to the 2012 machine. I attached a few screenshots of what i see in the vsphere client, nothing seems maxed out, the cpu in the vm doesn't go that high, no disk IO problems in ressource monitor, memory usage seems ok.. but god the vm is sluggish.

I'm installing microsoft exchange 2013 CU2 right now on it, i started the install yesterday 3:00 PM, it' been 16 hours and it's not done. Installing office 2010 took me about 1 hour also.

This vm is a production server, it's a small business email server. I got to get this thing working in 2 hours.. at least i took a snapshot before i started working on it yesterday, because i didn't trust the VM, it's so slow.

Not so sure where to start, but hell if anyone could help in any way, i'd be quite happy. If someone offering consulting services could remote on this server here to find out wth is happening.. i'm open to any options..

Edit: I used hd speed to test the drive speed, and it's really inconsistent on the 2012 VM, compared to a server 2003 vm i got here.. So it might have to do with disk IO ? Not so sure.. Drives in the server are 600GO 15K SAS, 3 drives in raid5 on a H710 controller.

Edit 2: Here is a performance test of the network.. It's really not performing as it should http://i.imgur.com/3spPndJ.jpg

Thanks !!!!

EDIT :

Okay, i've opened up a ticket with VMWare on SnS. After a few hours, we ended up saying that microsoft server 2012 was the issue, because all the other vm's are running smooth.

I've installed a dummy server 2012 and it's giving me great performance, so there's something wrong in that particular vm that making it sluggish, but i can't figure out what !

Running the task manager eats up 20% of the cpu.. Just idle, the machine uses 50% of the cpu.. That's really not normal. Everything is eating up the cpu just too fast, for some reason.

VMWare had me reconfigure the vm's on the esxi host to use 2 vcpu's instead of 8 each, which was a better practice.. It gave a little bit of performance help on the hosts. But that 2012 machine is still sluggish.

Anyone has any clues ? I've updated the dell server from the lifecycle controller firmware to all the latest bios, raid controller firmware, everything has been updated.

Solution

Firstly, try to check for the latest updates of server BIOS/EFI/iDRAC/HBA firmwares, operating systems and so on. Apply all the necessary updates.

Secondly, try to look at performance charts of your host/guest which are available from vSphere client (real-time only without vCenter) and focus on

• do you have VMware tools installed in your guest? do you use paravirtualized SCSI adapter with your guest?

• host CPU usage - host -> performance tab -> advanced -> cpu - what is average CPU load on the host? is it over 75%? are there peaks over 90%? it may signal that host CPU is saturated.

• guest CPU usage - host -> select the vm -> performance tab -> advanced -> cpu - the same like above plus check the CPU READY TIME, is it over 2000ms? again, the host CPU may be saturated.

• guest memory usage - host -> vm -> performance tab -> advanced -> memory - if memory average is over 80% or peaks over 90% the guest may have higher memory demand and so it is another performance bottleneck.

• guest memory swapping/ballooning - host -> performance tab -> advanced -> memory - check swap-in/swap-out and balloon rates. if it is over zero the host is swapping or ballooning vm memory which may make the vm sluggish as it requires access to disks.

• storage overloaded - host -> performance tab -> advanced -> disk - check STOP DISK metric, if it is over 0 zero disks/LUNs may be overloaded. additionally, check read/write latencies, if it is over 10ms in average or over 20ms in peaks disks/LUNs are slow or overloaded.

The above statistics can be seen from ESXi CLI with esxtop tool as well. But it requires some experience and knowledge of various counters to interpret its output (this article is about esxtop and storage performance).

Further, I would recommend you to read this VMware knowledge base article about performance troubleshooting. Then, there is a plenty of useful whitepapers and guides regarding this topic, e.g. vSphere Monitoring and Performance or Performance Best Practices for vSphere 5.1 .

Comments

[Ryan Ries]

The VMware tools on the guest OS fully updated?

[ewwhite]

How much physical RAM in the server?

[xeon]

RAID5 is most likely your bottleneck, look into RAID10 for good performance with good protection.

[joeqwerty]

Are you saying that you tested the speed of the guest hard drive? If so, that doesn't strike me as providing information that will allow you to diagnose a bottleneck on the underlying disk array.

[cheebster]

Ryan Ries - Yes, latest build.

[cheebster]

joeqwerty - yes, i tested it on the guest os

[cheebster]

Oh.. And all the other VM's i have are snappy. Server 2008, 2003.. all is clean and fast. It's just that 2012 that is giving me headaches.. Also, it's crashing quite often with a BSOD refering to TCPIP.SYS. I've reconfigured the vm to use different flavors of network adapters since it was, maybe network related.. no change at all, still crashing. kinda desperate here.

[dsmsk80]

Did you test network performance of the machine? Please check this kb articles as well - kb.vmware.com/kb/1001805, kb.vmware.com/kb/2008925 . Especially, the 2nd one helped me a few times but it was with Windows Server 2008.

[cheebster]

I'll check these articles right away. I tried to post what im getting in esxtop but i can't format it properly in that comment box, it seems

[cheebster]

Okay, well it seems i've got RSS Enabled, host ready time is OK for the host, and i've selected high performance as the power plan, but that one didn't seem to change much.

[cheebster]

Server 2012 is fully updated.. I forgot to note that.

[dsmsk80]

Do you have another Windows Server 2012 virtual machine? Does it behave the same weird way? Did you check storage? What about disk read/write/highest latencies?

[cheebster]

Okay, did another test as asked, ive tried testing network performance From the 2012 vm to the 2008 vm, speed is 62mb write, 12mb read.. From the 2008 vm to the 2003 vm, speed is 536mb write and 346mb read. WAAAAY faster

[dsmsk80]

I would just create a dummy Windows 2012 virt machine to confirm this weird behavior and to avoid any other changes which might have been done on the current machine. Btw, you should have SnS purchased with vSphere Essentials license so you can try to contact VMware support as well.

[cheebster]

Installing one right now :)

[dsmsk80]

Check this comments at tinyurl.com/qyqhy6r as well. Was the machine rebooted after vmware tools installation?

[cheebster]

Reboots.. hundreds ! - what is SnS exactly ? Thanks.

[cheebster]

I'm wondering what i should do from now, either purchase a support incident from VMWare or keep messing with it.. The dummy 2012 vm i've installed works, it's better than the other but i wouldn't say it's snappy. Pretty hard for me to figure out whether or not it should be faster than it is.. Nothing's running on, no exchange or anything and i've just used 4gb ram on it. I'm not going anywhere...

[dsmsk80]

I would really recommend you to open a case with VMware support now as it seems there may be a bug how Windows 2012 guest with multiple vCPUs is scheduled/handled on ESXi hypervisor resulting in huge performance loss. Btw, you wrote that you tried to change network interfaces type. I'm guessing Windows 2012 has vmxnet3 as default card which is paravirtualized and may bring some problems with the latest operating systems versions. Did you try e1000e cards instead of it?

[cheebster]

Hi dsumsky. I've tried e1000e cards at first, then e1000 then vmxnet3 right now. All of them give the same result for me. I'm gonna open up a case with the vmware support on Monday and i'll post back here whatever happens. What's pretty sure is that this VM isn't behaving like it should !