socat reliable file transfer over TCP

Question

I am aware of disadvantages of "wait->stop" design, that usually is proposed with netcat:

server$ cat test.dat | nc -q 10 -l -p 7878
client$ nc -w 10 remotehost 7878 > out.dat

(It's not reliable : no meter how long you wait it's always possible that there was network bottleneck for one second longer. - Another thing -> why wait 10second if you can know immediately that data are transferred and start processing them !)

I'd like solution, with reliable and nice tcp stream closing.

I've found socat, with closing as described in man socat :

When one of the streams effectively reaches EOF, the closing phase begins. Socat transfers the EOF condition to the other stream, i.e. tries to shutdown only its write stream, giving it a chance to terminate gracefully.

I found following commands working:

Server sending file:

server$ socat -u FILE:test.dat TCP-LISTEN:9876,reuseaddr
client$ socat -u TCP:127.0.0.1:9876 OPEN:out.dat,creat

Server receiving file:

server$ socat -u TCP-LISTEN:9876,reuseaddr OPEN:out.txt,creat && cat out.txt
client$ socat -u FILE:test.txt TCP:127.0.0.1:9876

Is it reliable? Can it be improved? (Have I used right options? Are there better options to setup? - There are so many of them with socat)

Solution

Looks to me like your core is solid -- this should be reliable, and should quit once the file has been completely sent.

If out.txt already exists, though, then this setup might behave unexpectely. If out.txt is longer than test.txt, the last part of out.txt will remain, since socat is overwriting the file byte by byte instead of making sure the file is empty. There are a few ways to fix this, depending on what you want to do:

• OPEN:out.txt,creat,trunc will delete all the bytes in out.txt before writing to it. This option mimics what you'd expect from cp, and is probably what you want.
• OPEN:out.txt,creat,excl will refuse to write out.txt if it already exists. Use this option for extra safety.
• OPEN:out.txt,creat,append will append data to out.txt.

I also like to run md5sum on the source and destination files whenever I cobble something like this together, because of these sorts of corner cases.

Comments

[rogerdpack]

For followers, the second example ("server receiving file") there is the "typical" client sending a file to a server...also newer versions of netcat have a "-N -q 0" option which should result in more reliable transfers than the old "wait and hope" mechanism :)

[Grzegorz Wierzowiecki]

it's great you've mentioned about "trunc" and those edge cases. What about md5sum, in my case it's problem cause I need to transfer all stuff without closing connection ("one-shot" ;) ). Even thou, it's good to mention for other readers :).