Why is this tee losing stdout?

linux ssh bash pipe tee

6,501

Solution 1

Try this, an alternative way of doing the pipe that breaks:

#!/bin/bash
remote_ssh_account="depesz@localhost"
directory_to_tar=pgdata
nice tar cf - "$directory_to_tar" | \
    tee >(
        md5sum | \
        ssh "$remote_ssh_account" 'cat > /tmp/h3po4-MD5-2012-03-13.sum'
    ) > >(
        ssh "$remote_ssh_account" 'cat > /tmp/h3po4-data-2012-03-13.tar'
    )

Solution 2

Issue found. Here's the relevant portion of the strace:

[pid 10243] write(1, "pFl\r\347\345]\244Hi\336\253,-\231\247\344\234\241\332\302\252\315\243G\234\225+\241\323\316s"..., 4096 <unfinished ...>
[pid 10247] select(7, [3 4], [3], NULL, {10, 0} <unfinished ...>
[pid 10243] <... write resumed> )       = -1 EAGAIN (Resource temporarily unavailable)
[pid 10247] <... select resumed> )      = 1 (out [3], left {10, 0})
[pid 10243] write(2, "tee: ", 5tee:  <unfinished ...>
(...)
[pid 10243] write(2, "standard output", 15standard output <unfinished ...>
(...)
[pid 10243] write(2, ": Resource temporarily unavailab"..., 34: Resource temporarily unavailable) = 34

So, what's happening is that the remote ssh isn't yet ready for the write to continue. Most programs handle this correctly, but tee decides to die in a pile. See http://lists.freebsd.org/pipermail/freebsd-bugs/2012-February/047528.html for one reference to this sort of behavior. There are a couple others that I found too in a brief search for "EAGAIN tee".

The solution that lhunath found works because it effectively forces bash to handle the EAGAIN. Elegant.

6,501

Author by

Admin

Updated on September 18, 2022

Comments

Admin over 1 year
Simple script:
```
#!/bin/bash
remote_ssh_account="depesz@localhost"
directory_to_tar=pgdata
exec nice tar cf - "$directory_to_tar" | \
    tee >(
        md5sum - | \
        ssh "$remote_ssh_account" 'cat - > /tmp/h3po4-MD5-2012-03-13.tar'
    ) | \
    ssh "$remote_ssh_account" 'cat - > /tmp/h3po4-data-2012-03-13.tar'
```
Theoretically it should deliver the data and checksum, to remote machine.

But somehow the tee fails with:
```
tee: standard output: Resource temporarily unavailable
```
Did strace, but nothing came out of it. I see both ssh started, and tee writing to both of them, but only the pipe to ( md5sum | ssh ) gets data - strace of the ssh "data" doesn't get any data, and after 5 seconds tee shows the error.

Aside from this all works. 2 connections are established, tar works, md5sum and its delivery works.
- BMDan about 12 years
  
  On the strace front, try strace -fF. Works like a champ for me.
- Admin about 12 years
  
  -fF is not in manual, but there is -f, -F and -ff. I used strace -ff -o strace.log -s 512 ./z.sh
- BMDan about 12 years
  
  -fF is the same as -f -F. The -F is actually unneeded on most of the systems I work on, but it (tries to) follow vforks on older versions of strace, and has no effect on newer ones, but it doesn't hurt. -ff ends up writing a bunch of files (one per PID) that you then have to collate back into a sensible timeline, so I avoid it in the vast majority of situations. That said, -ff should work, but the output files will be oddly-named.
Admin about 12 years

Same error - you can also easily test it yourself - just change directory to tar (it shouldn't be too small), and remote_ssh_account - to something that you have passwordless access to.
BMDan about 12 years

Changed; please try again. Yes, I know that doesn't do exactly what you want, but I just want to try it. Please be sure you're running it the same way (i.e., as a script, not directly on the command line).
Admin about 12 years

This works, and delivers the file. While I do appreciate the help - tests like this can be ran everywhere - did my original script not fail for you?
Admin about 12 years

Seems to be working. I still would love to know why the | ssh thing breaks.
BMDan about 12 years

I can get the problem to reliably recur with dd if=/dev/urandom bs=1M count=20 | tee >(md5sum - | ssh user@host 'cat - > /tmp/foo.md5') | ssh user@host 'cat > /tmp/foo.tar'. Working on debugging why now.
BMDan about 12 years

It also happens with the order reversed (that is, with the cat>tar inside the >()). Neat. Adding the -i flag to tee (on a blind hunch) didn't help.
Admin about 12 years

Yeah - it looks that the pipe before final ssh breaks. removing it to use > >( ), as in lhunath example - helps, but also helps if I'll do: tee >( md5sum | ssh ) >( ssh ) > /dev/null Weird stuff.
BMDan about 12 years

Since the evolution of my answer can be confusing to read: I've updated my answer with the final analysis of why this failed in the first place.