What is limiting the # of simultaneous connections my ASP.NET application can make to a web service?

Question

I have an ASP.NET 4.0 application running atop IIS 7.5 on a 64-bit Windows Server 2008 R2 Enterprise machine with gobs of RAM, CPU, disk, etc.

With every web request, the ASP.NET application makes a connection to a backend web service (via raw sockets), which is running on the same machine.

Problem: There appears to be something limiting the # of simultaneous connections to the backend web service. Suspiciously, the number of concurrent connections is topping out at 16.

I found this key article from Microsoft explaining how to tweak IIS' settings to accomodate ASP.NET apps that make lots of web service requests: http://support.microsoft.com/?id=821268#tocHeadRef

I followed the article's recommendatinos, but still no luck. The setting that is particularly interesting is the maxconnection setting, which I even bumped to 999.

Any ideas what else could be throttling connections?

Note: When I cut IIS out of the mix and have the clients connect directly to the backend web service, it will happily open as many connections as I need, so I'm positive the backend is not the bottleneck. It must be something in IIS/ASP.NET-land.

Here's the relevant section of the machine.config which I'm sure is being read by the application (verified with appcmd.exe):

<system.web>
    <processModel autoConfig="false" maxWorkerThreads="100" maxIoThreads="100" minWorkerThreads="50" />
    <httpRuntime minFreeThreads="176" minLocalRequestFreeThreads="152"/>

    <httpHandlers />

    <membership>
        <providers>
            <add name="AspNetSqlMembershipProvider"
                type="System.Web.Security.SqlMembershipProvider, System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
                connectionStringName="LocalSqlServer"
                enablePasswordRetrieval="false"
                enablePasswordReset="true"
                requiresQuestionAndAnswer="true"
                applicationName="/"
                requiresUniqueEmail="false"
                passwordFormat="Hashed"
                maxInvalidPasswordAttempts="5"
                minRequiredPasswordLength="7"
                minRequiredNonalphanumericCharacters="1"
                passwordAttemptWindow="10"
                passwordStrengthRegularExpression="" />
        </providers>
    </membership>

    <profile>
        <providers>
            <add name="AspNetSqlProfileProvider" connectionStringName="LocalSqlServer" applicationName="/"
                type="System.Web.Profile.SqlProfileProvider, System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
        </providers>
    </profile>

    <roleManager>
        <providers>
            <add name="AspNetSqlRoleProvider" connectionStringName="LocalSqlServer" applicationName="/"
                type="System.Web.Security.SqlRoleProvider, System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
            <add name="AspNetWindowsTokenRoleProvider" applicationName="/"
                type="System.Web.Security.WindowsTokenRoleProvider, System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
        </providers>
    </roleManager>
</system.web>
<system.net>
    <connectionManagement>
        <add address="*" maxconnection="999"/>
    </connectionManagement>
</system.net>

Solution 1

Most of the answers provided here address the number of incoming requests to your backend webservice, not the number of outgoing requests you can make from your ASP.net application to your backend service.

It's not your backend webservice that is throttling your request rate here, it is the number of open connections your calling application is willing to establish to the same endpoint (same URL).

You can remove this limitation by adding the following configuration section to your machine.config file:

<configuration>
  <system.net>
    <connectionManagement>
      <add address="*" maxconnection="65535"/>
    </connectionManagement>
  </system.net>
</configuration>

You could of course pick a more reasonable number if you'd like such as 50 or 100 concurrent connections. But the above will open it right up to max. You can also specify a specific address for the open limit rule above rather than the '*' which indicates all addresses.

MSDN Documentation for System.Net.connectionManagement

Another Great Resource for understanding ConnectManagement in .NET

Hope this solves your problem!

EDIT: Oops, I do see you have the connection management mentioned in your code above. I will leave my above info as it is relevant for future enquirers with the same problem. However, please note there are currently 4 different machine.config files on most up to date servers!

There is .NET Framework v2 running under both 32-bit and 64-bit as well as .NET Framework v4 also running under both 32-bit and 64-bit. Depending on your chosen settings for your application pool you could be using any one of these 4 different machine.config files! Please check all 4 machine.config files typically located here:

• C:\Windows\Microsoft.NET\Framework\v2.0.50727\CONFIG
• C:\Windows\Microsoft.NET\Framework64\v2.0.50727\CONFIG
• C:\Windows\Microsoft.NET\Framework\v4.0.30319\Config
• C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Config

Solution 2

I realize the question might be rather old, but you say the backend is running on the same server. That means on a different port, probably other than the default port 80.

I've read that when you use the "connectionManagement" configuration element, you need to specify the port number if it differs from the default 80.

LINK: maxConnection setting may not work even autoConfig = false in ASP.NET

Secondly, if you choose to use the default configuration (address="*") extended with your own backend specific value, you might consider putting the specific value first! Otherwise, if a request is made, the * matches first and the default of 2 connections is taken. Just like when you use the section in web.config.

LINK: <remove> Element for connectionManagement (Network Settings)

Hope it helps someone.

Solution 3

Might it be possible that you're using a WCF-based web service reference? By default, the ServiceThrottlingBehavior.MaxConcurrentCalls is 16.

You could try updating your service reference behavior's <serviceThrottling> element

<serviceThrottling
    maxConcurrentCalls="999" 
    maxConcurrentSessions="999" 
    maxConcurrentInstances="999" />

(Note that I'd recommend the settings above.) See MSDN for more information how to configure an appropriate <behavior> element.

Solution 4

Have you tried to set the value of the static DefaultConnectionLimit property programmatically?

Here is a good source of information about that true headache... ASP.NET Thread Usage on IIS 7.5, IIS 7.0, and IIS 6.0, with updates for framework 4.0.

Solution 5

See the "Threading" section of this page: http://msdn.microsoft.com/en-us/library/ff647786.aspx, in conjunction with the "Connections" section.

Have you tried upping the maxconnection attribute of your processModel setting?

View more solutions

Comments

[Rob Sobers]

Yes, I have. But that article you cited points out something interesting that ever other article on the topic neglected to mention: that the maxconnection attribute does not apply to local web service calls. In this particular case the web service is local, but we have that same problem in another environment where the service is not local.

[Benjamin Pollack]

I wish we were, but we're not. The service being consumed is running in Apache/Python/mod_wsgi on another machine, and, as Rob mentioned, it's clearly not the issue.

[John Saunders]

Service References are on the client. How is your client consuming the Apache service?

[Ruben]

Like John said: the throttling is set on the client consuming the web service. Perhaps the phrase "your service behavior" is a bit misleading. Does it make more sense when phrased as "your service reference behavior"?

[Ahmad]

@RobSobers - I think that the above link is worthy of a second look. Check the part about minLocalRequestFreeThreads - This worker process uses this setting to queue up requests from localhost (where a Web application calls a Web service on the same server) if the number of available threads in the thread pool falls below this number. This setting is similar to minFreeThreads, but it only applies to requests that use localhost.

[Rob Sobers]

Yup, I covered that base. Thanks anyway, @BenSwayne! I verified that I modified the correct machine.config (64-bit 4.0).

[Rob Sobers]

I imagine this should not matter, but I'm going to try it anyway.

[Rob Sobers]

@Ahmad Yup, I set minLocalRequestFreeThreads as well (see my machine.config in the question.

[Benjamin Pollack]

@john It's consumed via a TcpClient (the service vends custom binary blobs, not WCF/SOAP or similar). Those configuration options seem to be purely impacting you if you use ServiceHost, not TcpClient; am I missing something?

[John Saunders]

Why not use "Add Service Reference"?

[BenSwayne]

@RobSobers: In this case I would be a little suspicious of the implementation code. Perhaps you are running out of threads or something? Can you throw your TcpClient webservice call code into a console app and see if you can achieve a better request rate? This will prove whether its an IIS specific configuration or a broader .NET configuration or your code.

[Uri Abramson]

Does this line go in the client machine?

[Zakos]

thank you , your settings combine with processModel twekaing from codeproject.com/Articles/133738/… fixed "ISAPI ‘C:\windows\Microsoft.Net\Framework\v2.0.050727\aspnet_isapi‌​.dll’ reported itself as unhealthy for the following reason: ‘Deadlock detected" problem

[vibs2006]

@BenSwayne does the same concept applies for SMTP connections as well? I am designing a bulk email sending application so will this ConnectionManagement property will be useful in sending bulk mails as well (using multi threading for mail.send function)?

[Brain2000]

This is how we fixed it a few years back. We had concurrency issues and this seemed to clear it up. Net.ServicePointManager.DefaultConnectionLimit = 1000 is what we used. You must set it BEFORE you create connection classes.