Why is my tar.gz file corrupt after using bash ftp to move file to a remote server?

Question

I back the tar.gz file up on the local server, then ftp it to a remote server. If I pull it to my desktop via ftp client from the local server, the file works fine. If I pull it to my desktop via ftp client from the remote server, it is a few bytes smaller and corrupt. I've tried using the client to force ascii and binary mode, and neither corrupted the file, so I don't see why that would be causing the corruption in the script. Any ideas? The file transfers fine, it just won't open. Thank you in advance.

cd $FSBACKDIR
ATTACH='for file in *$DATE.tar.gz; do echo -n -e "put ${file}\n"; done'
ftp -nv <<EOF
open $FTPHOST
user $FTPUSER $FTPPASS
cd $FTPDIR
$ATTACH
quit
EOF

Solution

To me, it still seems likely to be an ASCII vs. binary mode problem, despite the test you did with your desktop client. Your desktop client may be smarter than the command-line FTP client on the sending server (your local server that you're running your script on).

For example, if the local server is Windows (which uses CRLFs as line endings) and the remote server is Unix (which uses just LFs as line endings), and you're not specifying binary mode and your FTP software doesn't auto-detect it and do the right thing, then you'd be using ASCII mode for the transfer, which should strip the CRs off of any CRLF pairs. If your gzipped tarball happens to have the byte pattern 0x0d0a appearing in it anywhere, it would lose the 0x0d.

If the command-line FTP client on your sending system (I guess that's your local server) is anything like the one on my system, all you'd have to do to test this theory is to add the binary command before or after the cd line:

cd $FSBACKDIR  
ATTACH='for file in *$DATE.tar.gz; do echo -n -e "put ${file}\n"; done'  
ftp -nv <<EOF  
open $FTPHOST  
user $FTPUSER $FTPPASS  
binary  
cd $FTPDIR  
$ATTACH  
quit  
EOF  

One last thought: If it's not ASCII vs binary mode after all, I'd look to see if maybe the FTP ALG in a NAT gateway between your local and remote server (or between the remote server and your desktop) is somehow corrupting the file in transit. I suppose it could also be some other kind of proxy in between hosts, not specifically a NAT gateway's FTP ALG.

Comments

[Josh]

Thank you. @Spiff. I must have been putting 'binary' in the wrong place when I tried it earlier. I guess I needed to sleep on it and have someone tell me to continue to focus where I was already looking. Much appreciated.