File access error with FileSystemWatcher when multiple files are added to a directory

Question

I am running into an issue with a FileSystemWatcher when multiple files are placed into the watched directory. I want to parse the file as soon as it is placed in the directory. Typically, the first file parses fine, but adding a second file to the directory causes an access issue. Occasionally, the first file doesn't even parse. There is only one application running and watching this directory. Eventually, this process will be running on multiple machines and they will be watching a shared directory but only one server can parse each file as the data is imported into a database and there are no primary keys.

Here is the FileSystemWatcher code:

public void Run() {
  FileSystemWatcher watcher = new FileSystemWatcher("C:\\temp");
  watcher.NotifyFilter = NotifyFilters.FileName;
  watcher.Filter = "*.txt";

  watcher.Created += new FileSystemEventHandler(OnChanged);

  watcher.EnableRaisingEvents = true;
  System.Threading.Thread.Sleep(System.Threading.Timeout.Infinite);
}

Then the method that parses the file:

private void OnChanged(object source, FileSystemEventArgs e) {
  string line = null;

  try {
    using (FileStream fs = new FileStream(e.FullPath, FileMode.Open, FileAccess.Read, FileShare.None)) {
      using (StreamReader sr = new StreamReader(fs)) {
        while (sr.EndOfStream == false) {
          line = sr.ReadLine();
          //parse the line and insert into the database
        }
      }
    }
  }
  catch (IOException ioe) {
    Console.WriteLine("OnChanged: Caught Exception reading file [{0}]", ioe.ToString());
  }

When moving the second file, it is catching

System.IO.IOException: The process cannot access the file 'C:\Temp\TestFile.txt' because it is being used by another process.

I would expect to see this error if it was running on multiple machines, but it is only running on one server for now. There shouldn't be another process using this file - I have them created and copy them into the directory when the application is running.

Is this the proper way to set up the FileSystemWatcher? How can I see what has the lock on this file? Why doesn't it parse both files - do I have to close the FileStream? I want to keep the FileShare.None option because I only want one server to parse the file - the server that gets to the file first parses it.

Solution 1

A typical problem of this approach is that the file is still being copied while the event is triggered. Obviously, you will get an exception because the file is locked during copying. An exception is especially likely on large files.

As a workaround you could first copy the file and then rename it and listen to the renaming event.

Or another option would be to have a while loop checking whether the file can be opened with write access. If it can you will know that copying has been completed. C# code could look like this (in a production system you might want to have a maximum number of retries or timeout instead of a while(true)):

/// <summary>
/// Waits until a file can be opened with write permission
/// </summary>
public static void WaitReady(string fileName)
{
    while (true)
    {
        try
        {
            using (Stream stream = System.IO.File.Open(fileName, FileMode.Open, FileAccess.ReadWrite, FileShare.ReadWrite))
            {
                if (stream != null)
                {
                    System.Diagnostics.Trace.WriteLine(string.Format("Output file {0} ready.", fileName));
                    break;
                }
            }
        }
        catch (FileNotFoundException ex)
        {
            System.Diagnostics.Trace.WriteLine(string.Format("Output file {0} not yet ready ({1})", fileName, ex.Message));
        }
        catch (IOException ex)
        {
            System.Diagnostics.Trace.WriteLine(string.Format("Output file {0} not yet ready ({1})", fileName, ex.Message));
        }
        catch (UnauthorizedAccessException ex)
        {
            System.Diagnostics.Trace.WriteLine(string.Format("Output file {0} not yet ready ({1})", fileName, ex.Message));
        }
        Thread.Sleep(500);
    }
}

Yet another approach would be to place a small trigger file in the folder after copying is completed. Your FileSystemWatcher would listen to the trigger file only.

Solution 2

I'd have left a comment above, but I don't have enough points yet.

The top-rated answer to this question has a block of code that look like this:

using (Stream stream = System.IO.File.Open(fileName, FileMode.Open, FileAccess.ReadWrite, FileShare.ReadWrite))
{
    if (stream != null)
    {
        System.Diagnostics.Trace.WriteLine(string.Format("Output file {0} ready.", fileName));
        break;
    }
}

The problem with using FileShare.ReadWrite setting is that it is requesting access to the file basically saying "I want to read/write to this file, but others can also read/write to it." This approach failed in our situation. The process that was receiving the remote transfer did not put a lock on the file, but it was actively writing to it. Our downstream code (SharpZipLib) was failing with the "file in use" exception because it was trying to open the file with a FileShare.Read ("I want the file for reading, and only let other processes read as well"). Because the process that had the file open was already writing to it, this request failed.

However, the code in the response above is too relaxed. By using FileShare.ReadWrite, it was succeeding in obtaining access to the file (because it was asking for a Share restriction that could be honored), but the downstream call continued to fail.

The share setting in the call to File.Open should be either FileShare.Read or FileShare.None, and NOT FileShare.ReadWrite.

Solution 3

When you open the file in your OnChanged method, you're specifying FileShare.None, which according to the documentation, will cause any other attempts to open the file to fail while you've got it open. Since all you (and your watcher) are doing is reading, try using FileShare.Read instead.

Solution 4

Simple solution would be to dispose the filesystemwatcher once you recieve the notification. before copying the file, make the current thread wait till it recieves the filesystemwatcher disposed event. then you can continue copying the changed file without access problems. I had same requirement and i did it exactly like what i mentioned. it worked.

Example Code:

public void TestWatcher()
{
    using (var fileWatcher = new FileSystemWatcher())
    {

        string path = @"C:\sv";
        string file = "pos.csv";

        fileWatcher.Path = path;
        fileWatcher.NotifyFilter = NotifyFilters.CreationTime | NotifyFilters.LastWrite;
        fileWatcher.Filter = file;

        System.EventHandler onDisposed = (sender,args) =>
        {
           eve.Set();
        };

        FileSystemEventHandler onFile = (sender, fileChange) =>
        {
           fileWatcher.EnableRaisingEvents = false;
           Thread t = new Thread(new ParameterizedThreadStart(CopyFile));
           t.Start(fileChange.FullPath);
           if (fileWatcher != null)
           {
               fileWatcher.Dispose();
           }
           proceed = false;
        };

        fileWatcher.Changed += onFile;
        fileWatcher.Created += onFile;
        fileWatcher.Disposed+= onDisposed;
        fileWatcher.EnableRaisingEvents = true;

        while (proceed)
        {
            if (!proceed)
            {
                break;
            }
        }
    }
}

public void CopyFile(object sourcePath)
{
    eve.WaitOne();
    var destinationFilePath = @"C:\sv\Co";
    if (!string.IsNullOrEmpty(destinationFilePath))
    {
        if (!Directory.Exists(destinationFilePath))
        {
            Directory.CreateDirectory(destinationFilePath);
        }
        destinationFilePath = Path.Combine(destinationFilePath, "pos.csv");
    }           

    File.Copy((string)sourcePath, destinationFilePath);
}

Solution 5

FileSystemWatcher fires watcher.Created event two times for every single file creation 1ce when file copy is started and 2nd time when file copy is finished. All you have to do is ignore 1st event and process event the second time.

A simple example of event handler:

private bool _fileCreated = false;
private void FileSystemWatcher_FileCreated(object sender, FileSystemEventArgs e)
{
    if (_fileCreated)
    {
        ReadFromFile();//just an example method call to access the new file
    }

    _fileCreated = !_fileCreated;
}

View more solutions

Comments

[Tai Squared]

These test files are tiny - only a few lines of short text, so it shouldn't be locked for too long during the copy. How would I loop to check if the file is ready to write to?

[Gishu]

Thanks! I'm ashamed I couldn't think of a while loop as a solution to this - I needed to read an updated version of an output file of another app. I got multiple events per write - so I used a timer that is reset each time and waits for X secs before triggering the code that reads the updated file. But then I missed updates if the app wrote twice within X secs.

[Oskar Kjellin]

+1 But you should really have a max time to wait so that the process isn't locked forever

[joedotnot]

No need for FileAccess.ReadWrite, or FileShare.ReadWrite; I used FileAccess.Read, FileShare.None and it's working fine, i.e. if you can get an exclusive lock to read the file, then you know the other process has finished creating the file and released it.

[Pablo Costa]

@joedotnot you solution didn't worked for me. Even setting FileAccess.Read and FileShare.None , i still get issues about file being used by another process.

[HackSlash]

If you are processing multiple files and you dispose of the filesystem watcher, you are going to have a bad time.