Windows Service application hangs

Question

Background:
- Running Window Server 2008 R2.
- The servers has been patched with all the recent updates.
- Server is running 5 services built in .NET 3.5 and all of these services use timers to check database on repeated basis (mostly every 10 seconds).
- These services are not CPU/RAM intensive.
- Server doesn't have any performace and resources issues or bottlenecks.

For most of the time, everything works as expected, but from time to time, some (or all) services simply stop working. I'm logging all the app exceptions to file, but there are none when it comes to failure. There is also no error in the event logger and the Services manager treats the services as running. I have to stop the services and start them once again to restore the functionality.

This behavior is unpredictable, sometimes it takes a week or a month before it stops working. Also, sometimes the services "dies" all together or only some of them at the same time.

Only thing that crossed my mind is the Timer object. I've been using the System.Timers.Timer and found several forum threads stating it is unreliable as the garbage collector may free the instance. I've tried retaining it using GC.KeepAlive() to no avail. I've followed a few suggestion on moving System.Timers.Timer to System.Threading.Timer but this didn't make any difference either.

Right at the moment, I'm desperate to find out the source of this behavior. Is there any known similar issue? How can I debug it when there is not exception raised and the event log is silent too?

Thank you for any advice that may lead to any solution.

UPDATE: Including the bare-boned code of the current state:

private System.Threading.Timer timerPublish = null;
private bool timerDelegateMethodRunning = false;

protected override void OnStart(string[] args)
{
    SetupTimer();
}

protected override void OnStop()
{
    if (timerPublish != null)
    {
        timerPublish.Dispose();
    }
}

public void SetupTimer()
{
    if (timerPublish != null)
    {
        timerPublish.Dispose();
    }
    TimerCallback callbackMethod = new TimerCallback(this.timerPublish_Elapsed);
    timerPublish = new System.Threading.Timer(callbackMethod, null, 5000, 5000);
}

void timerPublish_Elapsed(Object stateInfo)
{
    if (timerDelegateMethodRunning)
    {
        return;
    }
    timerDelegateMethodRunning = true;

    try
    {
        // Processing code here
    }
    finally
    {
        timerDelegateMethodRunning = false;
    }
}

UPDATE2: Thank you guys for your insights and advices. I'll try to debug the services on the production server once the issue happens again. I'll report back as soon as I have anything new (probably in several weeks).

Solution 1

Why do you complicate things? :) Just use the Timer.Change() method to trigger the timer again when you are ready.

Also know that any uncaught exceptions in the WorkerMethod will f*ckup your service.

public class YourService
{
    private System.Threading.Timer _timer;

    protected override void OnStart(string[] args)
    {
        //run once in 5 seconds.
        _timer = new System.Threading.Timer(WorkerMethod, null, 5000, Timeout.Infinite);
    }

    protected override void OnStop()
    {
        if (_timer != null)
        {
            _timer.Dispose();
            _timer = null;
        }
    }

    void WorkerMethod(object state)
    {
        // Processing code here

        _worker.Change(5000, Timeout.Infinite); //Run again in 5 seconds
    }
}

Update

I saw that you where using System.Timers.Timer. The biggest problem with it is that it ignores exceptions. That is, if your code throws an exception and you do not catch it: you will never be aware of that exception. It might very well be your problem.

Solution 2

Never draw a conclusion too soon before you know what is the cause of the hangs. There can be various unbelievable factors, but dump analysis or live debugging can tell you the truth,

http://blogs.msdn.com/b/tess/archive/2006/10/16/net-hang-debugging-walkthrough.aspx

If you like, you can even open a support case via http://support.microsoft.com