Copy a file over serial connection

Question

So what I am trying to achieve seems quite simple, but I can not seem to find a way to do it without using external utilities or not pre-installed libraries.

I have an embedded system running Linux (kernel 3.x depending on the models). On it I have access to most GNU core utilities (like cat, sed, ls, tar and so on).
I can access a shell session on it through /dev/ttyUSBx, for example using screen :

screen /dev/ttyUSB0 9600 

The computer from which I am running screen (let's call it source) is an ArchLinux install, which has access to the internet.
The embedded system (let's call it target) I am trying to access is a Linux machine without any network access, only a serial port which is /dev/ttyUSBx.
I need to copy a binary file to a specific location on the target, which is able to then flash some hardware (in this case an FPGA) automatically through a cron job.
I can not really install softwares on the target, only making small shell scripts.
Do you know of any POSIX shell way I can use to have the same behaviour as scp, but going through the /dev/ttyUSBx interface instead of ssh?

I thought of using pipes and file redirections to:
- cat the binary to /dev/ttyUSBx from the source.
- And on the target, to redirect the stream to the file.
I have no idea of how to even begin. The project I need to use this for is due tomorrow and I am lost right now, any idea would be welcome, very much preferably standard POSIX tools I can use directly from a shell script.

I do have access to gcc on the target, in case the solution needs to be compiled, but I would still prefer a shell solution, since compilation on the target is awfully slow, and would require more effort to then deploy to multiple machines.

My goal is to automate this workflow (reduce user interaction as much as possible). Maybe something like mounting the target filesystem as an NFS?

Solution 1

One possibility is to encode the binary file as ASCII text, either using the traditional uuencode, or the slightly more moden base64. The programs uuencode and uudecode are provided by the package sharutils, and base64 is in coreutils. It seems base64 is more likely to already be present on a modern GNU/Linux distribution.

The result of encoding the binary file is a large amount of text, which can in principle be copy/pasted over the terminal connection into the decoder on target. With a lot of data, using the clipboard can be impractical, but we can use facilities in screen to paste the contents of a file into the terminal.

1. On source, run base64 FILE > FILE.b64.
2. In the screen session connected to target, type Ctrl-A :readreg p /path/to/FILE.b64. (If your screen control key is something other than Ctrl-A, type that instead.) It seems you have to provide a complete path to FILE.b64; ~ doesn't work. Screen should report something like "Slurped 26665052 characters into buffer".
3. In the console on target, run base64 -d > FILE.
4. Type Ctrl-A :paste p.
5. Type Ctrl-D.

Another long-standing way to deal with this issue is ZModem, a venerable method to transfer files over terminal connections. Support for ZModem is built into many terminal emulators, such as Konsole so long as the lrzsz package is installed. However, chances are lrzsz is not installed, so using base64 would be better.

These methods all address the problem of transferring a file over a serial link which you are using at the time for a console. If you don't need the console, it should be simpler to arrange for target to dump anything from the link into a file; but that will be problematic if the serial console is your only way to interact with the device!

Solution 2

Based on everyone's responses so far, and considering the installed utilities on the target, I found a solution that works by executing a script from the source only. The script goes as follows:

dest="/dev/$(dmesg | grep "now attached to ttyUSB" | awk '{print$NF}' | tail -1)"
cat binfile | gzip | base64 > binfile.64.gz
md5=$(md5sum binfile | awk '{print $1}')
printf "\necho \"[ \\\$(md5sum <target-dir>/binfile | awk '{print \\\$1}') == %s ] &&\\ 
<flashing commands>\" > <target-dir>/flash.sh\n" \
    "$md5" \
    > "$dest"
printf " echo \"%s\" | base64 -d | gunzip > <target-dir>/binfile &&\\
/bin/sh <target-dir>/flash.sh
" "$(cat binfile.64.gz)" \
    > "$dest"

It sends a flashing program that checks the checksum of the source's and the target's binfiles are the same
compresses and encodes the binfile,
sends it,
and uses the flashing program to flash the card. If anything breaks in the encoding/decoding, it should not flash the target. Any other problem is not likely to cause problems to the system as a simple reboot will restore it to its default state.

Here is a link to my full implementation of that process:
https://github.com/Soulthym/cycloneV-serial-flasher

Comments

[Jim L.]

Possible duplicate: unix.stackexchange.com/questions/460342/…

[Mark Plotnick]

Does busybox rx on the embedded system output any message?

[Soulthym]

@MarkPlotnick busybox: command not found sadly

[Mark Plotnick]

Uucp is an optional part of POSIX and will do what you want, but it's rare to find it installed on a base system.

[Soulthym]

@MarkPlotnick would have been a perfect solution but it is not installed sadly..

[Mark Plotnick]

If you can install uucp on the Arch Linux system, you can use the cu program to send ascii-encoded files.

[Soulthym]

Just did, but I can't seem to find the correct syntax for uucp. Following the manpage, when I uucp file /dev/ttyUSB0!/home/root/file bash says that an event was not found (probably related to the ! character)

[Alessio]

do you have the ppp package installed on your embedded device (and on your "source" computer)? if so, you can use it to create a point-to-point network connection between "source" and "target" - and run scp, sftp, ssh, etc over that.

[Alessio]

Alternatively, you could use SLIRP. It's packaged for debian and probably other distros, and the source code is available at slirp.sourceforge.net to compile for your "target" machine. I haven't used SLIP (or SLIP) since the very early 1990s, but it was a great way of getting an IP connection from just a direct null-modem serial or dialup shell login account before PPP accounts were commonly available.

[Mark Plotnick]

You can put a backslash in front of the ! to quote it. And uucp will probably require some configuration file editing before it or cu can be used.

[Soulthym]

uu(en|de)code is not installed, but base64 seems to be! That's a good start, but now I need to automate the copy and paste process... Don't you think it is possible to > it to /dev/ttyUSBx and from the embedded system to cat < /dev/ttywhatever > file.64?

[Nick Matteo]

Yes, it's possible, but the terminal currently running on the serial line complicates things. The questions in the related list, such as unix.stackexchange.com/questions/229984/…, show how to do this with minicom which is probably better than screen for this sort of thing.

[Soulthym]

the binary file seems to be way too big for the clipboard, so copy/paste aren't a solution in this case sadly

[Soulthym]

an alternative I have found is to cat bin | gzip | base64. Apparently I can read it from /dev/ttyS0, I just need to save it to a file, decode it and gunzip it! Should be straightforward!

[Nick Matteo]

@Soulthym, I figured out how to get screen to paste the contents of a file into the current console session, which should work to avoid the clipboard issue. (see updated answer)