Checking if file is completely written

Question

Currently I'm working on the project that does processing files from source directory in one of it's routines. There's a Java process that's looking for specified directory and tries to read and process files if they exist. Files are quit large and updates by other thirdparty process. The question is how can I check if the file is completely written? I'm trying to use file.length() but looks like even if writing process hasn't been completed it returns actual size. I have a feeling that solution should be platform dependent. Any help would be appreciated.

UPDATE: This question is not really different from the duplicate but it has an answer with working code snippet that is highly rated.

Solution 1

Does the producer process close the file when its finished writing? If so, trying to open the file in the consumer process with an exclusive lock will fail if the producer process is still producing.

Solution 2

I got the solution working:

private boolean isCompletelyWritten(File file) {
    RandomAccessFile stream = null;
    try {
        stream = new RandomAccessFile(file, "rw");
        return true;
    } catch (Exception e) {
        log.info("Skipping file " + file.getName() + " for this iteration due it's not completely written");
    } finally {
        if (stream != null) {
            try {
                stream.close();
            } catch (IOException e) {
                log.error("Exception during closing file " + file.getName());
            }
        }
    }
    return false;
}

Thanks to @cklab and @Will and all others who suggested to look in "exclusive lock" way. I just posted code here to make other interested in people use it. I believe the solution with renaming suggested by @tigran also works but pure Java solution is preferable for me.

P.S. Initially I used FileOutputStream instead of RandomAccessFile but it locks file being written.

Solution 3

One simple solution I've used in the past for this scenario with Windows is to use boolean File.renameTo(File) and attempt to move the original file to a separate staging folder:

boolean success = potentiallyIncompleteFile.renameTo(stagingAreaFile);

If success is false, then the potentiallyIncompleteFile is still being written to.

Solution 4

I don't think there is a general solution for that. Looking for a file size is wrong as some applications can set file size prior any write call. One of the possibility is to use locking. This will require that writer accrue a write lock ( or exclusive lock ). If you can't modify writer, then you may use tools provided by OS, like fuser on Linux to see is there a process which still accesses the file.

Solution 5

If you plan to use this code on a single platform, you may be able to use NIO's FileLock facility. But read the documentation carefully, and note that on many platforms, the lock is only advisory.

Another approach is to have one process write the file with a name that your process won't recognize, then rename the file to a recognizable name when the write is complete. On most platforms, the rename operation is atomic if the source and destination are the same file system volume.

View more solutions

Comments

[cklab]

Have you considered using file locks?

[Viktor Stolbin]

Thanks. Checking this. Can we be sure locks are always created by writing process?

[cklab]

Well, the notion is if you attempt to obtain a lock to it, you should get an exception. I just came across this as well: docs.oracle.com/javase/7/docs/api/java/io/… Never used it but you can give it a shot to see how it works out. Looks interesting.

[Matthieu]

I don't know why that question is marked as duplicate: the answer in the other question is when you have control of the writing process.

[Viktor Stolbin]

Files are copied using rsync command under linux so I believe it does. Nice idea, will try it now.

[Viktor Stolbin]

Good point. Thanks. Will do it if no success with java. It's a bit hard to change the producer behavior due some circumstances.

[Nilesh]

This works on Winows but not on Linux.

[Viktor Stolbin]

@Nelish this is weird because I initially implemented it on Linux for Linux

[Nilesh]

Well I tried it on UNIX. A winscp or ftp was transferring file while I tried to lock and I was able to!

[Viktor Stolbin]

@Nilesh this might be a case if transferring process doesn't acquire exclusive lock, I don't know details. It worked with rsync for me.

[rkosegi]

rsync is using temporary file and renames it to final once finished, that's why it worked.

[Daniel Rodríguez]

I've tried this solution in Windows and works fine, but if the source file is in "Read Only" mode, it will never process it. Is there a solution for that case?

[Avik]

Solution should be platform agnostic.

[hasnain_ahmad]

Could you write an example to illustrate the procedure?

[Erwin Bolwidt]

Good suggestion for Windows users, but it's not platform-agnostic. On Linux/Unix you can delete and rename files that are still open by other processes.

[JoshDM]

That would be why my answer says "with Windows" and not "for all systems".

[ccallendar]

This solution works on some versions of windows/java, but newer versions seem to allow reading and writing to a file even if it's being written by other process (file.canWrite() will be true while a file is being copied). In this case I used the file.renameTo(file) to check if it's still being written.

[john ktejik]

this is not a very helpful correct answer :(