cpu usage increasing up to 100% in infinite loop in thread

Question

I am implementing a web based chat platform in ASP.NET Web Application, and I use technique similar to long polling. I mean I keep each web request from client for a specific time period(timeout) or until new message arrives, and then response is sent to the client.

I keep connected clients in memory(dictionary object) and when ever new message is sent to a client, I write this message into receiver client's messages array. Client needs to send a request to get his own messages, and I keep this request in an array in memory.

I am using asynchronous http handler for listenings client request, I am keeping web requests in an array in memory. I use threads to check for new messages continously from memory (in dictionary which is created for each client).

I do not use .net thread pool threads to check for new messages or timed out web requests.I create threads like this:

System.Threading.Thread t = new Thread(new ThreadStart(QueueCometWaitRequest_WaitCallback));
t.IsBackground = false;
t.Start();

In each thread's QueueCometWaitRequest_WaitCallback method I am in an infinite while loop:

while (true)
{
...
Thread.Sleep(100);
}

In this method, I am checking for web request time out or new message for each Web Request which is also kept in an array in memory.

Everything was working good until I noticed that CPU usage is reaching up to 100% in time. (in minutes after the first connected client) At the beginning of first request everything seems to be normal, I mean the CPU usage is not higher than 10% while returning a response to the client. But in time even with 2 clients the CPU usage is increasing up to 100%. It seems CPU usage is 100% only when writing to a response for a client request. If no client is left then everything return to a normal (CPU usage is about 0%) until new web request is done by a client.

I don't know the threads in detail, but I am suspicious about the new threads which I created and works infinitely. It is like operating system gives them more CPU usage and resource in time since they are working all the time, and this Thread.Sleep(100) is not working.

Here is the QueueCometWaitRequest_WaitCallback() method:

void QueueCometWaitRequest_WaitCallback()
{
   while (true)
   {
      if (processRequest.Length == 0)
      {
          Thread.Sleep(100);
      }
      else
      {
          for (int i = 0; i < processRequest.Length; i++)
          {
               Thread.Sleep(100);

               // below I am checking for new message or request time out 
               .................
               .................

               // If new message or time out I write to response
          }
      }    
   }
}

I hope I could explain the situation, and I am open to any suggestion as well (like implementing in a different way)

If you can help me with this problem I will appreciate gratefully, Thanks

Solution

Just as a general best-practices comment as opposed to direct answer - it's not advisbable to write a Thread.Sleep(100) inside your message receiver thread. A better method would be to use Thread.Join as previously mentioned or ManualResetEvent wait handles. For instance, you could code like this:

private ManualResetEvent waitHandle;
private object syncRoot = new object();
private bool isRunning = false;

void CreateThread()
{
    this.waitHandle = new ManualResetEvent(false);

    isRunning = true; // Set to false to kill the thread
    System.Threading.Thread t = new Thread(new ThreadStart(QueueCometWaitRequest_WaitCallback));         
    t.IsBackground = false; 
    t.Start();
}

void PushData()
{
    // On incoming data, push data into the processRequest queue and set the waithandle
    lock(syncRoot)
    {
        processRequest.Add(/* ... your data object to process. Assumes this is a queue */);
        waitHandle.Set(); // Signal to the thread there is data to process
    }
}

void QueueCometWaitRequest_WaitCallback() 
{    
    while (isRunning)    
    {       
        // Waits here using 0% CPU until the waitHandle.Set is called above
        this.waitHandle.WaitOne();

        // Ensures no-one sets waithandle while data is being processed and
        // subsequently reset
        lock(syncRoot)
        {
            for (int i = 0; i < processRequest.Length; i++)           
            {                        
                // Process the message. 
                // What's the type of processRequest? Im assuming a queue or something     
            }       

            // Reset the Waithandle for the next requestto process
            this.waitHandle.Reset();
        }
    }        
} 

This would ensure that your thread uses 0% CPU while waiting and only consumes CPU when there is work to do.

Failing that have you thought about a third party solution to asynchronous bi-directional messaging? I have used RabbitMQ (AMQP) with great success in .NET applications to handle high throughput messaging. The API for RabbitMQ means you get an event back when a message has been received which can then be processed on a background thread.

Best regards,

Comments

[Mehmet]

Thanks for your answer. But then another question arrives: I need to return a response to the client even there is no new message for a given time out period. In this case how can I waitHandle.Set();? I mean how can I know to start threads?

[Dr. Andrew Burnett-Thompson]

No probs. Just was thinking - you shouldn't get 100% CPU from your code example. How many threads are you creating? (Just a hunch). should be one of course! It's not one per request is it?

[Mehmet]

Not one per request, Total of 5 threads. Requests are in an array

[Dr. Andrew Burnett-Thompson]

@Mehmet I would seriously recommend looking in to RabbitMQ as a third party asynchronous messaging solution. This will handle message pipes between point to point (one client, one server), multi-cast (server to all clients) with low CPU usage, high throughput. You could implemement a periodic status update to all clients by implementing a Timer on the server which if no new notifications were received, sent a message to all clients.

[Mehmet]

Thanks again, I will look RabbitMQ, but unfortunately we have our web site open and I don't know if I can change implementation totally into RabbitMQ

[Dr. Andrew Burnett-Thompson]

@Mehmet ok fair enough. Have you used any profilers to determine where the bottleneck in code is? As I said your code doesnt suggest to me it should cause 100% CPU. Try Jetbrains dotTrace or Redgate Ants Profiler

[Mehmet]

@ Dr. Andrew Burnett-Thompson Thanks, I changed code to use the System.Threading.Timer with a period of 150ms and now CPU is not spiking up to 100%. Can you explain it why? Unfortunately timer is using the thread pool which I don't want :(

[Dr. Andrew Burnett-Thompson]

@Mehmet not sure. The only explanation I can offer is you are creating more than 5 threads as you say. 5x threads operating with 100ms sleeps should not cause the CPU to spike! The Timer will use the threadpool so will cap creation of threads. Can you post more code? Also try the profilers I mentioned. Not sure if they work with ASP.NET - dotTrace is your best bet. Fantastic tools to diagnose performance problems.

[Mehmet]

I checked the code and found I was creating 10 threads (System.Threading.Thread). And the while loop is shown above. Maybe you are write, it was the number of threads causing this problem. Is it any possibility to use CPU higher each time in an infinitely loop which sleeps for a very small time(100ms)?

[Dr. Andrew Burnett-Thompson]

@Mehmet even 10x threads at 100ms sleep shouldn't max out your CPU. Something else is wrong. Try using a System.Timers.Timer with 100ms elapsed to poll the request array then if there are requests dispatch the work onto a (single) separate thread. This ensures the threadpool is used for the brief operation of checking the incoming requests and your thread is used for the longer running operation of processing the response. To decouple threads from one another you can use Queue<T> and Add action delegates or an object describing work items to perform. You could then use the waitHandle also.

[Dr. Andrew Burnett-Thompson]

@Mehmet finally I would suggest either revising your question with updated data or creating a new one as the Question and Answer are starting to no longer accurately reflect the problem. Best regards,