C++ WinApi: ReadDirectoryChangesW() Receiving Double Notifications

Question

I try to understand ReadDirectoryChangesW function so I can be effectively informed on change of content in several directories (files overwritten, files deleted, renamed, etc..).

One of my recent observations is, that for every file write operation, I receive always two notifications for single file.

I traced that very carefully, and I am sure, that if I overwrite a file (say a .txt file with a new content - basically couple of extra letters inside), ReadDirectoryChangesW() notifies me two times per that file save.

This is serious thing, as I expect to be notified only once per change. I do not wish to unintentionally repeat operations that should happen only once in my application.

Is this behavior known? Is there a way to receive only one notification per one change, please? Is there way to effectively avoid double notifications?

I use:

• Unmanaged C++
• Visual Studio 2012
• Windows 7 x64

I use pretty basic code to do my tests, but you will want to see it, so here it is:

HANDLE hDir = CreateFile(
    lpDir,
    FILE_LIST_DIRECTORY,
    FILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE,
    NULL, 
    OPEN_EXISTING, 
    FILE_FLAG_BACKUP_SEMANTICS, 
    NULL);

    int nCounter = 0;
    FILE_NOTIFY_INFORMATION strFileNotifyInfo[1024];
    DWORD dwBytesReturned = 0;   

    while(TRUE)
    {
        if( ReadDirectoryChangesW ( hDir, (LPVOID)&strFileNotifyInfo, sizeof(strFileNotifyInfo), FALSE, FILE_NOTIFY_CHANGE_LAST_WRITE, &dwBytesReturned, NULL, NULL) == 0)
        {
            ErrorCheck(_T("Reading Directory Change"));
        }
        else
        {
            _tcout << _T("File Modified: ") << strFileNotifyInfo[0].FileName << endl;
            _tcout << _T("Loop: ") << nCounter++ << endl;
        }
    }

Solution 1

ReadDirectoryChangesW() has a very myopic view of the file system. It sees every change to the file system and dutifully reports them. And yes, there is often more than one when you write to a file. It is an implementation detail of the particular file system you are using, but any common one in Windows also keeps a directory entry for a file that stores metadata for the file.

So you see the write for the file data. But you also see it changing the directory entry. In particularly the file size, likely to change when you write a file and add data to the file. And the last-write and last-access timestamps recorded in the directory entry. The api is otherwise blind to the kind of change being made, it only sees the low-level write. It is also completely unaware of what particular process asked for the write.

This is something you will have to deal with, there is no way to distinguish these writes. All that you know is "the file was changed". How, why, by whom and how often is entirely undiscoverable.

Something else you will have to deal with is that, at the time the notification is generated, the process that writes the file is very likely to still have a lock on the file. Which prevents you from doing anything useful with the file yourself. Like reading the file or copying it is likely to fail. You have to wait until the process is done with the file and has closed its handle to the file. There's no way to discover this, other than by trying to open the file yourself and deny any sharing. This requires a timer, periodically trying to acquire a lock on the file yourself. Once you got that plumbing in place, getting more than one change notification for the file doesn't matter anymore.

Solution 2

This is the result operation of the running code simultaneously with the procmon logging after file save in notepad. There are two notifies from ReadDirectoryChangesW() and two notifies from procmon. 2 IRP_MJ_WRITE 1 from Notepad (WriteFile) 1 from System Cache Manager (CcWriteBehind)

Comments

[TheUndeadFish]

First you say every write operation, but then your example is overwrite. I would speculate that overwriting could involve more than a single operation from the directory's perspective (such as delete then create) as opposed to just writing/appending to an existing file. But I don't have any authoritative reference on this...

[Raymond Chen]

possible duplicate of Filesystemwatcher double entries

[Bunkai.Satori]

@TheUndeadFish, hi and thank you for your idea. It is indeed very clever. I tried my tests with another flags. I can say, that for example FILE_NOTIFY_CHANGE_ATTRIBUTES returns two notifications as well. This can be explained, that on file overwrite, more than one attribute is normally changed?

[Nik Bougalis]

This seems reasonable but I'm not sure... the documentation for FILE_NOTIFY_CHANGE_LAST_WRITE at msdn.microsoft.com/en-us/library/windows/desktop/… states "Any change to the last write-time of files in the watched directory or subtree causes a change notification wait operation to return. The operating system detects a change to the last write-time only when the file is written to the disk. For operating systems that use extensive caching, detection occurs only when the cache is sufficiently flushed."

[Bunkai.Satori]

@sergmat: +1 halo and thanks for your explanation. Your analysis is great and perfectly explains this issue. I am just sorry, i was not able to find answer on my own. It looks, that Process Monitor from Sysinternals.com is very powerful tool.

[Bunkai.Satori]

@NikBougalis, hi Nik, yes, I have read that paragraph from the documentation. It looks that Sergmat explained that, and I will have to monitor all the notifications based on time elapsed, so i will avoid those doubles.

[Bunkai.Satori]

+1 and thank you for your answer. You explained that in very nice way, with proposed idea for solution. I will implement timer based monitoring. Your's and and Sergmat's answers are both very good. It is difficult to decide which one to mark as the Accepted Answer, but, as you provide additional info, i am marking your answer. Thanks again.

[Nik Bougalis]

His supposition may be correct but I don't know that he explained it. The behavior he observes runs contrary to what the documentation states.

[sergmat]

There is nothing to explain, in the screenshot all is visible.This is the result operation of the running code simultaneously with the procmon logging after file save in notepad. There are two notifies from ReadDirectoryChangesW() and two notifies from procmon.