Java 7 WatchService - Ignoring multiple occurrences of the same event
Solution 1
I had a similar issue - I am using the WatchService API to keep directories in sync, but observed that in many cases, updates were being performed twice. I seem to have resolved the issue by checking the timestamp on the files - this seems to screen out the second copy operation. (At least in windows 7 - I can't be sure if it will work correctly in other operation systems)
Maybe you could use something similar? Store the timestamp from the file and reload only when the timestamp is updated?
Solution 2
WatcherServices reports events twice because the underlying file is updated twice. Once for the content and once for the file modified time. These events happen within a short time span. To solve this, sleep between the poll()
or take()
calls and the key.pollEvents()
call. For example:
@Override
@SuppressWarnings( "SleepWhileInLoop" )
public void run() {
setListening( true );
while( isListening() ) {
try {
final WatchKey key = getWatchService().take();
final Path path = get( key );
// Prevent receiving two separate ENTRY_MODIFY events: file modified
// and timestamp updated. Instead, receive one ENTRY_MODIFY event
// with two counts.
Thread.sleep( 50 );
for( final WatchEvent<?> event : key.pollEvents() ) {
final Path changed = path.resolve( (Path)event.context() );
if( event.kind() == ENTRY_MODIFY && isListening( changed ) ) {
System.out.println( "Changed: " + changed );
}
}
if( !key.reset() ) {
ignore( path );
}
} catch( IOException | InterruptedException ex ) {
// Stop eavesdropping.
setListening( false );
}
}
}
Calling sleep()
helps eliminate the double calls. The delay might have to be as high as three seconds.
Solution 3
One of my goto solutions for problems like this is to simply queue up the unique event resources and delay processing for an acceptable amount of time. In this case I maintain a Set<String>
that contains every file name derived from each event that arrives. Using a Set<>
ensures that duplicates don't get added and, therefore, will only be processed once (per delay period).
Each time an interesting event arrives I add the file name to the Set<>
and restart my delay timer. When things settle down and the delay period elapses, I proceed to processing the files.
The addFileToProcess() and processFiles() methods are 'synchronized' to ensure that no ConcurrentModificationExceptions are thrown.
This simplified/standalone example is a derivative of Oracle's WatchDir.java:
import static java.nio.file.StandardWatchEventKinds.ENTRY_CREATE;
import static java.nio.file.StandardWatchEventKinds.ENTRY_DELETE;
import static java.nio.file.StandardWatchEventKinds.ENTRY_MODIFY;
import static java.nio.file.StandardWatchEventKinds.OVERFLOW;
import java.io.IOException;
import java.nio.file.FileSystems;
import java.nio.file.FileVisitResult;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.SimpleFileVisitor;
import java.nio.file.WatchEvent;
import java.nio.file.WatchKey;
import java.nio.file.WatchService;
import java.nio.file.attribute.BasicFileAttributes;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Iterator;
import java.util.Map;
import java.util.Timer;
import java.util.TimerTask;
public class DirectoryWatcherService implements Runnable {
@SuppressWarnings("unchecked")
static <T> WatchEvent<T> cast(WatchEvent<?> event) {
return (WatchEvent<T>)event;
}
/*
* Wait this long after an event before processing the files.
*/
private final int DELAY = 500;
/*
* Use a SET to prevent duplicates from being added when multiple events on the
* same file arrive in quick succession.
*/
HashSet<String> filesToReload = new HashSet<String>();
/*
* Keep a map that will be used to resolve WatchKeys to the parent directory
* so that we can resolve the full path to an event file.
*/
private final Map<WatchKey,Path> keys;
Timer processDelayTimer = null;
private volatile Thread server;
private boolean trace = false;
private WatchService watcher = null;
public DirectoryWatcherService(Path dir, boolean recursive)
throws IOException {
this.watcher = FileSystems.getDefault().newWatchService();
this.keys = new HashMap<WatchKey,Path>();
if (recursive) {
registerAll(dir);
} else {
register(dir);
}
// enable trace after initial registration
this.trace = true;
}
private synchronized void addFileToProcess(String filename) {
boolean alreadyAdded = filesToReload.add(filename) == false;
System.out.println("Queuing file for processing: "
+ filename + (alreadyAdded?"(already queued)":""));
if (processDelayTimer != null) {
processDelayTimer.cancel();
}
processDelayTimer = new Timer();
processDelayTimer.schedule(new TimerTask() {
@Override
public void run() {
processFiles();
}
}, DELAY);
}
private synchronized void processFiles() {
/*
* Iterate over the set of file to be processed
*/
for (Iterator<String> it = filesToReload.iterator(); it.hasNext();) {
String filename = it.next();
/*
* Sometimes you just have to do what you have to do...
*/
System.out.println("Processing file: " + filename);
/*
* Remove this file from the set.
*/
it.remove();
}
}
/**
* Register the given directory with the WatchService
*/
private void register(Path dir) throws IOException {
WatchKey key = dir.register(watcher, ENTRY_CREATE, ENTRY_DELETE, ENTRY_MODIFY);
if (trace) {
Path prev = keys.get(key);
if (prev == null) {
System.out.format("register: %s\n", dir);
} else {
if (!dir.equals(prev)) {
System.out.format("update: %s -> %s\n", prev, dir);
}
}
}
keys.put(key, dir);
}
/**
* Register the given directory, and all its sub-directories, with the
* WatchService.
*/
private void registerAll(final Path start) throws IOException {
// register directory and sub-directories
Files.walkFileTree(start, new SimpleFileVisitor<Path>() {
@Override
public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs)
throws IOException
{
if (dir.getFileName().toString().startsWith(".")) {
return FileVisitResult.SKIP_SUBTREE;
}
register(dir);
return FileVisitResult.CONTINUE;
}
});
}
@SuppressWarnings("unchecked")
@Override
public void run() {
Thread thisThread = Thread.currentThread();
while (server == thisThread) {
try {
// wait for key to be signaled
WatchKey key;
try {
key = watcher.take();
} catch (InterruptedException x) {
return;
}
Path dir = keys.get(key);
if (dir == null) {
continue;
}
for (WatchEvent<?> event: key.pollEvents()) {
WatchEvent.Kind<?> kind = event.kind();
if (kind == OVERFLOW) {
continue;
}
if (kind == ENTRY_MODIFY) {
WatchEvent<Path> ev = (WatchEvent<Path>)event;
Path name = ev.context();
Path child = dir.resolve(name);
String filename = child.toAbsolutePath().toString();
addFileToProcess(filename);
}
}
key.reset();
} catch (Exception e) {
e.printStackTrace();
}
}
}
public void start() {
server = new Thread(this);
server.setName("Directory Watcher Service");
server.start();
}
public void stop() {
Thread moribund = server;
server = null;
if (moribund != null) {
moribund.interrupt();
}
}
public static void main(String[] args) {
if (args==null || args.length == 0) {
System.err.println("You need to provide a path to watch!");
System.exit(-1);
}
Path p = Paths.get(args[0]);
if (!Files.isDirectory(p)) {
System.err.println(p + " is not a directory!");
System.exit(-1);
}
DirectoryWatcherService watcherService;
try {
watcherService = new DirectoryWatcherService(p, true);
watcherService.start();
} catch (IOException e) {
System.err.println(e.getMessage());
}
}
}
Solution 4
I modified WatchDir.java to receive only human-made modifications. Comparing .lastModified()
of a file.
long lastModi=0; //above for loop
if(kind==ENTRY_CREATE){
System.out.format("%s: %s\n", event.kind().name(), child);
}else if(kind==ENTRY_MODIFY){
if(child.toFile().lastModified() - lastModi > 1000){
System.out.format("%s: %s\n", event.kind().name(), child);
}
}else if(kind==ENTRY_DELETE){
System.out.format("%s: %s\n", event.kind().name(), child);
}
lastModi=child.toFile().lastModified();
Solution 5
Here is a full implementation using timestamps
to avoid firing multiple events:
import java.io.File;
import java.io.IOException;
import java.nio.file.*;
import java.nio.file.attribute.BasicFileAttributes;
import java.util.HashMap;
import java.util.Map;
import static java.nio.file.LinkOption.NOFOLLOW_LINKS;
import static java.nio.file.StandardWatchEventKinds.*;
public abstract class DirectoryWatcher
{
private WatchService watcher;
private Map<WatchKey, Path> keys;
private Map<Path, Long> fileTimeStamps;
private boolean recursive;
private boolean trace = true;
@SuppressWarnings("unchecked")
private static <T> WatchEvent<T> cast(WatchEvent<?> event)
{
return (WatchEvent<T>) event;
}
/**
* Register the given directory with the WatchService
*/
private void register(Path directory) throws IOException
{
WatchKey watchKey = directory.register(watcher, ENTRY_MODIFY, ENTRY_CREATE, ENTRY_DELETE);
addFileTimeStamps(directory);
if (trace)
{
Path existingFilePath = keys.get(watchKey);
if (existingFilePath == null)
{
System.out.format("register: %s\n", directory);
} else
{
if (!directory.equals(existingFilePath))
{
System.out.format("update: %s -> %s\n", existingFilePath, directory);
}
}
}
keys.put(watchKey, directory);
}
private void addFileTimeStamps(Path directory)
{
File[] files = directory.toFile().listFiles();
if (files != null)
{
for (File file : files)
{
if (file.isFile())
{
fileTimeStamps.put(file.toPath(), file.lastModified());
}
}
}
}
/**
* Register the given directory, and all its sub-directories, with the
* WatchService.
*/
private void registerAll(Path directory) throws IOException
{
Files.walkFileTree(directory, new SimpleFileVisitor<Path>()
{
@Override
public FileVisitResult preVisitDirectory(Path currentDirectory, BasicFileAttributes attrs)
throws IOException
{
register(currentDirectory);
return FileVisitResult.CONTINUE;
}
});
}
/**
* Creates a WatchService and registers the given directory
*/
DirectoryWatcher(Path directory, boolean recursive) throws IOException
{
this.watcher = FileSystems.getDefault().newWatchService();
this.keys = new HashMap<>();
fileTimeStamps = new HashMap<>();
this.recursive = recursive;
if (recursive)
{
System.out.format("Scanning %s ...\n", directory);
registerAll(directory);
System.out.println("Done.");
} else
{
register(directory);
}
// enable trace after initial registration
this.trace = true;
}
/**
* Process all events for keys queued to the watcher
*/
void processEvents() throws InterruptedException, IOException
{
while (true)
{
WatchKey key = watcher.take();
Path dir = keys.get(key);
if (dir == null)
{
System.err.println("WatchKey not recognized!!");
continue;
}
for (WatchEvent<?> event : key.pollEvents())
{
WatchEvent.Kind watchEventKind = event.kind();
// TBD - provide example of how OVERFLOW event is handled
if (watchEventKind == OVERFLOW)
{
continue;
}
// Context for directory entry event is the file name of entry
WatchEvent<Path> watchEvent = cast(event);
Path fileName = watchEvent.context();
Path filePath = dir.resolve(fileName);
long oldFileModifiedTimeStamp = fileTimeStamps.get(filePath);
long newFileModifiedTimeStamp = filePath.toFile().lastModified();
if (newFileModifiedTimeStamp > oldFileModifiedTimeStamp)
{
fileTimeStamps.remove(filePath);
onEventOccurred();
fileTimeStamps.put(filePath, filePath.toFile().lastModified());
}
if (recursive && watchEventKind == ENTRY_CREATE)
{
if (Files.isDirectory(filePath, NOFOLLOW_LINKS))
{
registerAll(filePath);
}
}
break;
}
boolean valid = key.reset();
if (!valid)
{
keys.remove(key);
if (keys.isEmpty())
{
break;
}
}
}
}
public abstract void onEventOccurred();
}
Extend the class and implement the onEventOccurred()
method.
Related videos on Youtube
Sotirios Delimanolis
Couple posts on Medium: When you think you found a bug in TCP, don’t get cocky, kid. The other side of Stack Overflow content moderation
Updated on August 14, 2021Comments
-
Sotirios Delimanolis over 2 years
The javadoc for
StandardWatchEventKinds.ENTRY_MODIFY
says:Directory entry modified. When a directory is registered for this event then the WatchKey is queued when it is observed that an entry in the directory has been modified. The event count for this event is 1 or greater.
When you edit the content of a file through an editor, it'll modify both date (or other metadata) and content. You therefore get two
ENTRY_MODIFY
events, but each will have acount
of 1 (at least that's what I'm seeing).I'm trying to monitor a configuration file (
servers.cfg
previously registered with theWatchService
) that is manually updated (ie. through command linevi
) with the following code:while(true) { watchKey = watchService.take(); // blocks for (WatchEvent<?> event : watchKey.pollEvents()) { WatchEvent<Path> watchEvent = (WatchEvent<Path>) event; WatchEvent.Kind<Path> kind = watchEvent.kind(); System.out.println(watchEvent.context() + ", count: "+ watchEvent.count() + ", event: "+ watchEvent.kind()); // prints (loop on the while twice) // servers.cfg, count: 1, event: ENTRY_MODIFY // servers.cfg, count: 1, event: ENTRY_MODIFY switch(kind.name()) { case "ENTRY_MODIFY": handleModify(watchEvent.context()); // reload configuration class break; case "ENTRY_DELETE": handleDelete(watchEvent.context()); // do something else break; } } watchKey.reset(); }
Since you get two
ENTRY_MODIFY
events, the above would reload the configuration twice when only once is needed. Is there any way to ignore all but one of these, assuming there could be more than one such event?If the
WatchService
API has such a utility so much the better. (I kind of don't want to check times between each event. All the handler methods in my code are synchronous.The same thing occurs if you create (copy/paste) a file from one directory to the watched directory. How can you combine both of those into one event?
-
Sotirios Delimanolis almost 11 yearsThe
for
loop actually just loops once. The while loop loops twice with a single event inwatchKey.pollEvents()
, so I don't think this will work. -
Robert H almost 11 yearsIf you expand out your for each loop, you can get access to the size() method of the
pollEvents
list:List<WatchEvent<?>> events = watchKey.pollEvents(); System.out.println(events.size());
Does this also return 1, or does it accurately show 2 events? The count of watchEvent should be 2 if its a duplicate, but in this instance I think your 2 may reflect better in the list. -
Sotirios Delimanolis almost 11 years
pollEvents().size()
is one, buttake()
happens twice (then blocks again, waiting). It's confusing to talk about this. There are two OS events (ex: modify content and modify metadata) but only one human event. I guess theWatchService
API sees it as twoWatchKeys
with oneWatchEvent
each. -
Sotirios Delimanolis almost 11 years
WatchService
is a Java 7 feature, so yes, in that sense. If you copied over a file, you should see both anENTRY_CREATED
andENTRY_MODIFY
. -
Jayan almost 11 years@ Sotirios Delimanolis: You mean you get create notification when watching for StandardWatchEventKinds.ENTRY_MODIFY? Solution from tofarr seems to be a good workaround.
-
Sotirios Delimanolis almost 11 yearsYea, nvm, you aren't watching for it. I'm surprised you only get one
ENTRY_MODIFY
though. Tofarr's solution is what I've done for now, but I'm still giving it some time for other possible solutions. -
2rs2ts about 10 yearsWhat if the file was modified two or more times in that group of events? Hypothetically you'd have 4, 6, 8... etc. modify events, but the value of
.lastModified()
would only represent the latest event. -
Sotirios Delimanolis over 9 yearsDoesn't the
WatchService
key these events in the background? What would delaying retrieving them do? -
mmdemirbas over 9 yearsDoing so, you will receive an event with count value 2 instead of two separate events with count value 1. So, yo don't need to eliminate duplicate events manually. I walk this wa. Happy :)
-
Sotirios Delimanolis almost 9 yearsCan you explain what the code does? Does it combine both of those into one event? Does it use a solution where only one event is triggered?
-
FaNaJ almost 9 years@SotiriosDelimanolis in the method
isInterested
we are ignoring occurrence of same events that occurred within100 ms
. but in some cases it could be unsafe. for example, if two applications are modifying the same file and applicationB
finishes it's job50 ms
after than applicationA
did ... -
Florian Sesser almost 9 yearsThe OP:
I kind of don't want to check times between each event
-
Volceri over 7 yearsI had the same issue, then I read Jayan's comment regarding the notepad++ updating the file attributes twice. The problem is not the watcher, it was the file editor, in my case.
-
igracia about 7 years+1 for
Observable
. The only issue with this solution is that it's not very scalable, as you are spawning one thread per observer. That's a lot if you are going to watch several hundreds of files! -
Florian Sesser about 7 years@igracia you're very right, thanks! My use case was watching a single file, hadn't thought about scalability. FaNaJ's solution below probably works a lot better when watching a lot of files.
-
igracia about 7 yearsI guess you could combine both somehow, and have the
FileChangeNotifier
register anObservable
representation of a file change. Just have the while-loop in the solution you pointed out notifying the observers, and you have the best of both worlds ;-)Issuing a random, platform-dependantsleep
seems very prone to chance. -
Florian Sesser about 7 yearsI cluttered my implementation already with the
ignoreNext
flag for when I myself modified the file (a bit quick and dirty...). It should be relatively easy to drop that and add a Map of Paths to a List of Observers. I rather would like to not do it myself without a use case though, and remembering what an effort testing this on different platforms was... -
tresf over 6 yearsThis seems OK if 2 is guaranteed. In my tests, different applications behave differently.
cmd
andjava
trigger one event on write where asnotepad.exe
andnotepad++
trigger two. -
user2729516 over 6 yearsYou are right. This was not that helpful in some cases so I ended up using this solution from Dave Jarvis. May be you can use both the solutions combined if that helps you.
-
tresf over 6 yearsI've done some benchmarking. On my machine, saving a file with 1 line results in about 6ms between "duplicate" events. Saving a file with 10,000,000 lines results in about 500ms between "duplicate" events.
-
user2729516 over 6 yearsThe duplicate calls actually depend on the underlying file system and the application which changes the file content. The behavior changes from application to application as well so your mileage may vary.
-
nverbeek about 6 yearsThis is the best answer, thank you for the description of why there are two events.
-
Faizan Mubasher over 4 yearsNot working. Do I need to write it in separate thread? Currently I am writing in
main
method just for testing. -
Player1 about 4 yearsYou do need to write this in a separate thread. Works perfectly. I tested this in windows and had 3 modify entry events when overwriting a file. Added Thread sleep and problem was solved. Thank you for the answer.
-
Lei Yang almost 2 yearscan you also paste definition for
isListening
?