All of my scheduled tasks are failing with error code 0xE0434352 on Windows Server 2008 R2

Question

I have many scheduled tasks on a Windows Server 2008 R2 box, running a few different programs with various command line options. Last night, they all started failing with the last run result of 0xE0434352, which is the exception code for all .NET exceptions.

The applications are .NET applications so ordinarily I would suspect a coding issue. However, running the jobs manually from the command line works fine, and I confirmed there have been no binary or job changes in weeks.

What's my next step to see what's going on?

Solution

The origin of the problem was I had one scheduled task that was set to kill itself after 24 hours and then restart. It was a PowerShell script that started another program, call it the logger. The logger wasn't killed when the scheduled task died, and over time a large number of loggers built up on the server, though they were no longer writing to their output.

While there was no issue with processor utilization or memory, as the loggers used very little memory and weren't using any processor, it seems the Task Scheduler was holding on to a reference to them, and after a threshold was passed, all of the scheduled tasks stopped running with the above error.

Killing the loggers allowed task scheduler to keep running, and fixing the job so that the loggers were shut down cleanly solved the problem long-term.

Comments

[Greg Askew]

Are you using log4net?

[dsolimano]

@GregAskew I'm not using log4net, it's a process that captures traffic on an internal message bus and writes some of it to files.