IIS 7 worker process bottleneck, large number of awaiting requests under application pool ASP.NET 3.5 + 2.0

Question

I am using ASP.NET 2.0, .NET 2.0 Framework, and IIS 7. I am seeing a large queue of "requests" appear under the "worker process" option. State recorded appear to be Authenticate Request and Execute Request Handles more than anything else.

I have amended aspnet.config in C:\Windows\Microsoft.NET\Framework64\v2.0.50727 (32 bit path and 64 bit path) to include:

maxConcurrentRequestsPerCPU="50000"
maxConcurrentThreadsPerCPU="0"
requestQueueLimit="50000"

I have amended machine.config in C:\Windows\Microsoft.NET\Framework64\v2.0.50727\CONFIG (32-bit and 64-bit path) to include:

autoConfig="false"
maxIoThreads="100"
maxWorkerThreads="100"
minIoThreads="50"
minWorkerThreads="50"
minFreeThreads="176"
minLocalRequestFreeThreads="152"

Still I get this issue.

The issue manifests itself as a large number of requests in the Worker Process queue.

The number of current connections to the website display 500 when this issue occurs. I don't think I have seen concurrent connections over 500 without this issue occurring.

The web application slows as the requests block.

Refreshing the application pool resolves for a while (as expected) as the load is spread between the two pools.

The application pool in question FIXED REQUEST have been set to refresh on 50000.

Note: .NET 3.5 framework uses 2.0 framework appnet and machine configuration files, I believe.

Server resources (CPU, RAM) are not used to the full potential.

Solution 1

Ended up increasing the worker processes from 1 to 2 (web garden). Issue has not reoccurred since although were are using sessions but over the course of a month there have been no reports of session problems from end users.

EDITED TO ADD:

The specific issue was related to the generation of a extremely large CSV reports which are processed on the worker process server side, not a bug , just the way it works. HTML reports are fine, the XML transform is passed client side.

The separation of worker processes resolved the issues until the sever side xml transform to csv file creation can be passed to an addition process therefore removing any influence from other users. Was simply down to the size of the files / number of rows we were working with when trying to create CSV reports.

Solution 2

Can I suggest you take a look at the materials on Tess Ferrandez's site If It Is Broken, Fix It You Should.

What you want to be doing is capturing a dump of the worker process using ADPlus at the time you experience a high number of requests in the request queue and when the app starts to grind to a halt.

Once you grabbed this dump you want to load this into WinDBG+SOS and start tracking down the culprit.

Tess has a great series of labs about how to utilise these tools to great effect:

.NET Debugging Demos - Information and setup instructions

If you are able to, then you could also attach a profiler to the application (such as RedGate's Performance Profiler to try and discover the root cause.

Solution 3

Follow the KB article http://support.microsoft.com/kb/821268 and update me if you are still facing the issue.

You can also try FREB tracing in IIS 7. http://learn.iis.net/page.aspx/266/troubleshooting-failed-requests-using-tracing-in-iis-7/

Comments

[Dave Lawrence]

If you want the process model to use your values for maxIoThreads etc autoConfig should be set to false. Not true.

[scott_lotus]

Thanks for the post Dave, good spot. Amended.

[Dave Lawrence]

Glad to help... I've been navigating the murky waters of ASP tuning recently myself! :-)

[Lex Li]

Or simply @scott_lotus should contact Microsoft support via support.microsoft.com Tuning the settings can only remedy the hang if the threads are somehow independent of each other. But if the hang remains, it is almost obvious that you should pay attention to the threads themselves (deadlock or something similar).

[Kev]

@lex - yes he could "simply" contact support.microsoft.com, in fact we all should :). But then he'd also end up with a bill for their time and a few days of waiting for PSS to eventually tell him to fix his code. The only way he'll find out if he's got a deadlock is to examine the application using suitable diagnostic tools such as ADPlus+WinDBG+SOS. You can't start "thread tuning" until you know what the application is up to, and even then thread tuning probably isn't going to fix this if there is a deadlock.

[scott_lotus]

Thank you for the reply, re failed requests, sorry forgot to mention this has been configured. The resulting information was huge , very difficult to find anything that looked like an issue. Have also spent some time looking at Tez's post, again the resulting debugging file didnt offer anything relevant to me. Will examine again. Was expecting to see some form of error.

[scott_lotus]

In addition one of the aspect of the software is reporting. When a report is run it calls the a service running within the same application pool to find the address of each row. When the worker processes grow these address calls are the most prominant .. i.e so our app is exposed to slow downs cause by however large a user makes a report. Thats not the point thhough , the point is changing the environmental variables should allow more resources to be used but dont. i.e so i dont think its errors thats the problem, its the quanitiy of valid calls.

[Rahul Soni]

See if u can create a dump and host is somewhere from where I can download it. Will try to help.

[scott_lotus]

Sorry for delayed reply. Issue has raised its ugly head again when enabling a new feature that calls a table for a number of rows on 1) page load, 2) page refresh 3) 2 mins scheduled auto refresh on every page within an asp.net application. This feature (according to performance monitor) increases the number of total conncurrent connections to the IIS7 "web service" process. Always appears to hit the 600 + connecitons then bottlenecks and the connection log starts increasing. When turning off this new feature the problem clears.Not an IO prob ,everything else ok,other site , pool ok on server.

[scott_lotus]

Ps in process of opening up consultancy task with ISP & MS, will report back any findings if/when issue resolved.

[scott_lotus]

Trying the web garden approach (max worker proccess option from app ppol) , app seems to be happy with it so far after testing , all app features enabled , will see what happens of the next few days.

[scott_lotus]

The web garden approach has really helped , deployed 4 processes, when the bottleneck occurs it happens in a single process , from there is easy to identify the common REQUEST thread that is visble when the issue occurs. In this case it was a report caught in a loop due to data (in xml format) that the report could not process. I think we all simply assumed the data would always be correct not realising our INSERT process can do odd things.