LINUX: how to detect that ftp file upload is finished

Question

In my project I have a file uploading feature. Files are uploaded via FTP. I need to configure a listener that will check for new files and invoke a script only when file uploading is finished. Because if I run this script immediately after detecting the new file, it can start to process file that is not completely uploaded, which will cause an error. Can anybody tell if this is possible on LINUX and how can I do this?

Solution 1

I'd try using inotify, event code IN_CLOSE_WRITE.

Solution 2

Apache "Mina" ftp server (java) might be able to do what you want, including detecting a failed upload, as mentioned here

Quote:

From Ftplet.afterCommand, you should be able to look at the reply. For those failed transfers that FtpServer can detect (that causes an SocketException or IOException) this should be something like 426 or 551.

Ftplet overview here, including response codes.

The afterCommand method signature:

FtpletResult afterCommand(FtpSession session, FtpRequest request, FtpReply reply)

You'd check reply.getCode() in your overriden method. You should subclass DefaultFtplet rather than implementing Ftplet interface from scratch.

Note that DefaultFtplet::afterCommand shows how to detect what client command is being responded to. You can check for STOR or STOU and reply-code 426 or 551 to detect failed uploads.

However, this may not detect an upload intentionally terminated by the client, if the client app decides to treat the transfer as though the file was just shorter than it is. In the case of a unintentionally broken connection, I think the reply-code check will work. A test could be to kill the client app, or bring down the client machine's network interface.

To handle successful uploads (your original question), you can look for the success reply-code instead, ie 226.

Solution 3

Have a look at inotify

It doesn't automatically watch sub directories though, so if you need to monitor many ftp accounts (or the FTP client wants to create a sub dir and upload there) you'll need to handle this yourself.

Solution 4

I was looking for the same thing and stumbled on pureftpd which has an upload script feature. Sounds like exactly what was needed. Found the details here: http://www.linuxbyexamples.net/2012/10/config-ftp-server-trigger-upload-file-to-call-external-script.html

View more solutions

Comments

[Tim Post]

Can you specify what language you'd like to use?

[Tim Post]

+1 , but its worth mentioning that some ftp servers create a hidden temporary file to receive data until the xfer is finished. I.e. .foo-ftpupload-2837948723 in the same upload path as the target file , so you'll need to make sure your event loop can deal with that and not trigger the task until the file has actually been renamed.

[Tim Post]

Correct. A breadth-first search (similar to ftw(), but breadth-first) is recommended for whatever daemon is doing the watching so no sub directories are missed as it starts.

[andr111]

Thank you for suggestions. I tried inotify, but I found that it fires IN_CLOSE_WRITE in situation when file is not completely uploaded, but connection is lost (for example if I stop uploading or close FTP client). So I believe there is no way of detecting file COMPLETELY uploaded rather than using HTTP. Is this correct?

[mouhammed]

The event to look for in this case is close_write.