FTP module for Node.js that can sync files?

Question

I am looking for a module for Node.js - preferably available through npm that can sync files with an FTP server. The ftp module has all the basic functions, but writing a full blown sync system on top of it seems like a lot of work.

Solution 1

I'm the author of jsftp. It should be easy to sync through it simply walking through the directories you want to upload. Jsftp can handle parallel passive requests (internally they are still sequential, but you don't have to worry about it), so you could just use sequential ftp.put calls and it should work without much scripting effort on your side.

Even better and more idiomatic in node would be to use the yet to be documented ftp.getPutSocket method, which allows you to get the raw PASV socket and just push into it (using streams, for example). For a particular example of this case, you can look at how vfs-ftp does it.

Feel free to contact me if you need more help with implementing the syncing.

Solution 2

I did some research by searching npm, github and the web and couldn't come across any good looking solution to syncing.

I would simply start by writing my own (simple) synchronization lib by using some good ftp library. Personally I find jsftp to be the most appealing (http://sergi.github.com/jsftp/). Just combine jsftp with some filesystem walking - especially if you are fine with simple one-way syncing, this shouldn't be impossible to do.

Some ideas for implementing (one-way sync from local filesystem to ftp):

• Open connection to the ftp server
• Walk through local filesystem with fs.readdir
• Get stats from the corresponding ftp file and upload if local version is newer (Ftp.ls and Ftp.put if using jsftp)

Alternatively, ask yourself do you really need to perform the sync with node.js? For instance if you can run Python scripts in your system, you could simply use ftpsync2d, a bidirectional sync tool with ftp support (http://code.google.com/p/ftpsync2d/).

Solution 3

Take a look at node-ftpsync and grunt-ftpsync

It utilizes Sergi's jsftp library to intelligent sync files to a remote server.

Note: Currently, everything needed for push support is implemented except timestamp based update comparisons. For now updates are determined by comparing file sizes alone.

node-ftpsync provides both CLI support and a programming API. The preferred solution is to install it globally and run it in a directory containing a config.json. Alternately, it is also implemented as a lib that can be easily configured, extended, and launched within code.

grunt-ftpsync works the same way except configuration and running the application is done through grunt. It can even outputs fancy colorized logs.

Disclaimer: I am the author of node-ftpsync and grunt-ftpsync

Comments

[mzedeler]

Thanks for the suggestion. I actually started down the "I'll just whip up something" route first, and decided against it because it turned out a little too complex. The reason that I want to do it in node is to make it platform independent - it is for a Cakefile (coffeescript Makefile) that deploys to a local test server. The developers are using both Windows, Linux and Mac.

[jsalonen]

Thanks for the follow up. If you are using ftp for node app deployment, couldn't you do that directly, say, from a git repository? Also if you have shell access, you could use some deployment tool like deploy for automation (github.com/visionmedia/deploy)

[mzedeler]

Thanks for the suggestion - unfortunately, I'm in a firewalled environment where most usual things are considered unusual by our sysadmins. The server I deploy to is a Windows machine and we only have ftp access.

[mzedeler]

Thanks for the tip. I'll let you know if I get stuck.

[Evan Plaice]

+1 good answer. Using and established python project is definitely appealing but a good node implementation could also be integrated with grunt and used as a one-click-deploy solution. There are a few options available such as grunt-ftpdeploy but all of them do a wholesale push of everything. I created node-ftpsync and the grunt-ftpsync plugin because it can intelligently compare the local and remote trees to only sync the files that have changed.

[Evan Plaice]

+1 awesome library. It has proven to be very useful and robust. If you want to see this in action check out node-ftpsync. Push support is almost complete, I just need to finish the part that does timestamp comparisons based on MDTM.

[Vít Kotačka]

Thanks @EvanPlaice, that's very nice to hear. Let me know if you missed something in it or thought of any improvement to it!

[mzedeler]

One of my co-workers decided to reinvent the whole thing using a very simplistic approach that will only do one-way sync using a local state file. Its available here: npmjs.org/package/ftpkick

[Dr. Piyush Dholariya]

when I tried to upload any file using jsftp with method of jsftp.put or jsftp.getPutSocket I am getting this error events.js:72 < throw er; // Unhandled 'error' event < ^ < Error: read ECONNRESET < at errnoException (net.js:901:11) < at TCP.onread (net.js:556:19) I have tried all kind of approach with buffer, plain string evrything but its not working for me.

[Manwal]

@GiantSquid How would i found connection made successfully?

[wayofthefuture]

Is there a reason why timestamp support has not been implemented? Is it implemented in grunt-ftpsync?

[Evan Plaice]

@Dude2TheN grunt-ftpsync is nothing but a grunt-friendly wrapper for node-ftpsync. Timestamp support doesn't exist yet because the FTP protocol has weak or non-existent support for doing time comparisons and I haven't found a workable non-hacky solution to solve the problem yet.

[turmuka]

Hi, I have questions about the module. Is there a place where I can contact you in private? commenting is not convenient for this task. @GiantSquid