How to have a FileSystemWatcher thread run forever?

Question

I've hit this error when trying to run scipy.optimize.curve_fit() to fit a sinusoidal curve to data. Looking at other questions, I've tried increasing the value of maxfev all the way to a million and it still won't converge.

I also checked whether the array I have in for sigma (which isn't actually the error as we don't have that but a rough approximation) has any zeroes in it as that would cause it to trip up but it doesn't. It does hit a divide by zero error sometimes but I don't know what's causing it and I can't replicate it consistently.

This is the data I am fitting; it clearly has a sinusoidal trend and I've set the initial parameters to match the plot as well as I can.

It works when I have a smaller time range; it's only the larger datasets it gets stuck on.

This is the line of code:

params, params_covariance = optimize.curve_fit(test_func, array, avg_radon, p0=[naverage_true,nstd,np.pi/365,1], sigma = npercent,maxfev=10000)

Where the function to be fitted is

def test_func(x, P0, P1, P2, P3):
        return P0 + P1 * np.sin(P2*(x-P3))

and the initial parameters are the average of the function, the standard deviation (amplitude), a period of a year, and the shift I'm not really sure of. I've also tried initializing with the optimal parameters of the smaller dataset and no dice.

Again just to reiterate: I've tried increasing maxfev and checking sigma for zeroes, which are the solutions suggested in all the other questions of this form I can find.

Edit: I figured out that I get the divide by zero error only if I run the larger dataset directly after having run the smaller dataset. What could that mean?

Solution 1

If this is running on a server, a service seems like a good bet.

If you want something a little more interactive, I have a small application that lives in my system tray to show me when something happens in our Issues folder. It uses the Application.Run() method to keep itself going without the need for a loop, i.e.

class MainEntryClass
{
    static void Main(string[] args)
    {
        FolderAlerter fa = new FolderAlerter(); 
        Application.Run();
    }
}

Then I set up my FileSystemWatcher in FolderAlerter.

Solution 2

Basically, FileSystemWatcher is using some of the built-in hooks to subscribe to changes and just gets the OS to call your delegates when a change occurs. So all you have to do is stop your main thread from exiting, it doesn't have to actually do anything; By the time you get to that line, the main part of your program is over, it has done what it needed to do.

A simple way of doing what you want is to make this change to the sample from MSDN;

//while (Console.Read() != 'q') ;
System.Threading.Thread.Sleep(System.Threading.Timeout.Infinite);

This will put your main thread to sleep forever, but will still allow the delegates to fire and respond to changes in the directory. Note that it will keep running through multiple changes (just like the MSDN sample).

Note that the "while" in the MSDN sample doesn't really do much; You could replace that line with just "Console.Read()", which would then make any key quit the application. It's just a line to stop the main thread from exiting so that it is still around when the event notifications come in.

However, using the Sleep(Timeout.Infinite) leaves your app in a state where it cannot be easily stopped by a user, other than by killing the app. If you are writing this as a console app, why not stick with the "press Q to quit"? If you decide to write it as a service instead it will probably be OK, as long as you handle the shutdown/restart events etc.

Solution 3

To do this, you probably want to create a ManualResetEvent in a reset state and have your main thread WaitOne() on it. To terminate, some other callback can simply Set() the event which will cause WaitOne() to return.

Comments

[Not Sure]

The source code is actually for a quite old version. The IFS Kit at microsoft.com/whdc/devtools/ifskit/default.mspx provides updated source code (one of the examples it provides is the same as used by FileMon and the underlying ReadDirectoryChanges Win32 call).

[Jesse Weigert]

Application.Run() is a better method because it sets up the required message processing loop. I'm not sure, but I think this method will trigger the "application not responding" message when the users tries to shut down Windows.

[flytzen]

It didn't in my test - but I agree, it's not a good way to do it, hence my "However..." :)