IIS 7.5 stops serving requests for no apparent reason

Question

We're running a site on 4 virtual Win 2008 R2 64 bit servers. (On top of Hyper-V physical servers)

This is the only site on the IIS, and we use Windows Network Load Balancing to share the load between our 4 virtual servers.

We've used these virtual servers for approximately a week, and we're starting to see some issues.

For no apparent reason the IIS stops serving pages, and doesn't even respond with an error. So upon requesting a page from the server, the browser just waits infinitely (or until it decides to give up clientside)

Sometimes an iisreset fixes the issue, other times we have to reboot the entire virtual server.

There are no traces in the eventlog of why this happens, and there's no traces in our applications exception log neither.

Furthermore this happens even when there's a very small load on the server, so it doesn't seem to be because it's flooded with requests.

So frankly I'm at a loss here - I have no idea where to start debugging this issue :-(

I'm quite certain we never had these issues on our physical servers, however they were running Win 2003 32bit, so there are quite a few differences between them and the virtual ones. (Which obviously makes it difficult to tell what exactly causes this)

Solution

What kind of application is hosted on IIS 7?

If this is a .net Web site and if the pages are not working from the client machine even after hitting the server directly, I suggest yo remote desktop the server and browse the page directly from the server itself.

If you still not able to browse your Web site from the server itself, try browsing a simple .html page and see if that works. If .html page works, try accessing a .aspx page which does not make any database call and see if that works. If simple .aspx page does not work then there is some problem in the .net pipeline. Probably your requests are getting queued and hence the Web site does not respond.

Open Perfmon and add counters for ASP.NET object. Check the Request Queued and Request Execution Time, if you see any value in there then the answer is your requests are queued.
If they are not then taking a ETW trace will help which will give you some information on what's happening with your .aspx requests.

Check this for ETW tracing

Comments

[TomTom]

Damn, that one is heavy. Did you check system logs and can you rule out a NTLB issue?

[Steffen]

I did check system logs yes, there's absolute nothing there from the time the problem starts and until I reboot the system :-( I'm pretty certain I can rule out NTLB issues, as we've got the option of going straight to a specific server (ignoring the NTLB), and the result is the same.

[Steffen]

+1 Nice elaborate explanation, I'll follow your guidelines next time it happens :-)

[MikeT]

pleasure.. nice to be of any help :-)