"Unable to connect to remote server fail" in HttpWebRequest
Solution 1
Kev answered you question already, I just want to add that creating so many threads is not really good design solution (just context switching overhead is a big minus) plus it won't scale good.
The quick answer would be: use asynchronous operations to read data instead of creating a bunch of threads. Or at least use thread pool (and lower worker thread count). Remember that more connections to one source will only speed things up till some degree. Try benchmarking it and you will see that probably 3-5 connections will work faster that 2000 you are using now.
You can read more about asynchronous client/server architecture (IOCP - input/output completion ports) and its advantages here. You can start from here:
MSDN - Using an Asynchronous Server Socket
MSDN - Asynchronous Server Socket Example
CodeProject - Multi-threaded .NET TCP Server Examples
All of these examples uses lower level TCP object, but it can be applied to WebRequest/WebResponse as well.
UPDATE
To try thread pool version, you can do something like this:
ManualResetEvent[] events = new ManualResetEvent[ClientCount];
for (uint cnt = 0; cnt < events.Length; cnt++)
{
events[cnt] = new ManualResetEvent(false);
ThreadPool.QueueUserWorkItem(obj => PerformanceWorker());
}
WaitHandle.WaitAll(events);
Not tested, may need some adjustment.
Solution 2
I reckon you've maxed out the web site's application pool queue. The default is 1000 requests, you're flooding the server with 2000 requests more or less all at once. Increasing the timeout isn't going to solve this.
Try increasing the Queue Length for the application pool the site resides in.
You should try and capture the underlying HTTP status, that'll give you a clue as to what is really going on.
Update:
When I run your code and try and download a sizeable file (200MB) I get (503) Server Unavailable.
. Increasing the size of the Application Pool's request queue solves this (I set mine to 10000).
Only once did I see Unable to connect to remote server
and sadly have been unable to replicate. This error sounds like there's something broken at the TCP/IP layer. Can you post the full exception?
Solution 3
Go to Smart Thread Pool and downlod the code. It is an instance thread pool that constrains the number of threads. The .Net Thread pool can be problematic in applications that connect to web servers and SQL servers.
Change the loop to this
static void Main(string[] args)
{
var stp = new SmartThreadPool((int) TimeSpan.FromMinutes(5).TotalMilliseconds,
Environment.ProcessorCount - 1, Environment.ProcessorCount - 1);
stp.Start();
for (var i = 0; i < ClientCount; i++)
{
stp.QueueWorkItem(PerformanceWorker);
}
stp.WaitForIdle();
stp.Shutdown();
return;
}
This constrains the thread pool to use 1 thread per proc. Adjust this up until performance starts to degrade. Too many threads are worse than too few. you many find that this is optimal.
Also add this to you config. The value of 100 is a default I use. There is a way to do this in code but the syntax escapes me now.
<system.net>
<connectionManagement>
<add address=“*“ maxconnection=“100“ />
</connectionManagement>
</system.net>
George2
Updated on June 05, 2022Comments
-
George2 almost 2 years
I am using VSTS 2008 + C# + .Net 3.5 to develop a console application and I send request to another server (IIS 7.0 on Windows Server 2008). I find when the # of request threads are big (e.g. 2000 threads), the client will receive error "Unable to connect to remote server fail" when invoking response = (HttpWebResponse)request.GetResponse().My confusion is -- I have set timeout to be a large value, but I got such fail message within a minute. I think even if the connection are really larger than what IIS could serve, client should not get such fail message so soon, it should get such message after timeout period. Any comments? Any ideas what is wrong? Any ideas to make more number of concurrent connection being served by IIS 7.0?
Here is my code,
class Program { private static int ClientCount = 2000; private static string TargetURL = "http://labtest/abc.wmv"; private static int Timeout = 3600; static void PerformanceWorker() { Stream dataStream = null; HttpWebRequest request = null; HttpWebResponse response = null; StreamReader reader = null; try { request = (HttpWebRequest)WebRequest.Create(TargetURL); request.Timeout = Timeout * 1000; request.Proxy = null; response = (HttpWebResponse)request.GetResponse(); dataStream = response.GetResponseStream(); reader = new StreamReader(dataStream); // 1 M at one time char[] c = new char[1000 * 10]; while (reader.Read(c, 0, c.Length) > 0) { Console.WriteLine(Thread.CurrentThread.ManagedThreadId); } } catch (Exception ex) { Console.WriteLine(ex.Message + "\n" + ex.StackTrace); } finally { if (null != reader) { reader.Close(); } if (null != dataStream) { dataStream.Close(); } if (null != response) { response.Close(); } } } static void Main(string[] args) { Thread[] workers = new Thread[ClientCount]; for (int i = 0; i < ClientCount; i++) { workers[i] = new Thread((new ThreadStart(PerformanceWorker))); } for (int i = 0; i < ClientCount; i++) { workers[i].Start(); } for (int i = 0; i < ClientCount; i++) { workers[i].Join(); } return; } }
-
Kev over 14 yearsI've updated my answer, can you paste the full exception?
-
-
George2 over 14 yearsThanks! 1. I have increased it to 65535. I think you mean IIS Manager => Application pools => Advanced Settings => Queue Length. Is it the correct place and value you mean to set? 2. For underlying Http status, how to get it?
-
Lex Li over 14 yearsLike I commented on iis.net forum, HTTP error logs contain that information.
-
George2 over 14 yearsThanks Kev, for "Application Pool's request queue solves this", do you mean IIS Manager => Application pools => Advanced Settings => Queue Length?
-
George2 over 14 yearsAnother question is, what is the concurrent client # you are using? 2000?
-
George2 over 14 yearsTo @lextm-MSFT, I have posted to, appreciate if you could take a look, forums.iis.net/t/1161933.aspx
-
George2 over 14 yearsTo @Kev, my environment is using IIS bit rate throttling control extension for IIS 7.0. Could you reproduce this issue with IIS bit rate throttling control extension for IIS 7.0 enabled?
-
Kev over 14 years@George -Yes, IIS Manager => Application pools => Advanced Settings => Queue Length is the one I mean. I am running with ClientCount = 2000. What throttle rate settings are you using?
-
George2 over 14 yearsI keep the default throttling rate setting (I just select enable Bit Rate Throttling from IIS manager). Could you reproduce my reported issue? Maybe you can try 3,000 threads at client side. It should be easier to reproduce with 3,000 threads at client side.
-
George2 over 14 yearsFrom logs under C:\Windows\System32\LogFiles\HTTPERR, I got failed reasons of two categories, any ideas what is wrong? HTTP/1.1 GET /test.wmv - 1 Timer_MinBytesPerSecond DefaultAppPool HTTP/1.1 GET /test.wmv - 1 Connection_Dropped_List_Full DefaultAppPool
-
George2 over 14 yearsI want to simulate (doing performance testing) the scenario when there are large number of concurrent number of connections. Using thread pool could simulate this better than using real concurrent threads?
-
Audrius over 14 yearsYou should really try it yourself. I'm not sure if it will benefit you in this case (as this is for load testing, so not really performance important), but you will learn something new (-. Updated answer with thread pool example.
-
George2 over 14 yearsI have a related question here, appreciate if you could take a look. Thanks stackoverflow.com/questions/1600002/…
-
George2 over 14 yearsI have a related question here, appreciate if you could take a look. Thanks stackoverflow.com/questions/1600002/…
-
George2 over 14 yearsThe smart thread pool can not be opened. I searched a few on the web, do you mean this one? codeproject.com/KB/threads/smartthreadpool.aspx
-
George2 over 14 yearsWhy "The .Net Thread pool can be problematic in applications that connect to web servers and SQL servers."?