Bash - Redirect output to a variable or file descriptor, then read from the variable or file descriptor

Question

Here's an example of a bash script that redirects all output to a file (and shows output on the screen too):

# writing to it
exec > >(tee --ignore-interrupts ./log)
exec 2>&1

echo "here is a message"

# reading from it again
cat ./log

Now I don't want to create the file ./log. I want to keep all the stdout in memory, and be able to read from it again later in the script. Also I'm in a situation where there may be no root filesystem mounted.

I tried to do this with process substitution instead of ./log, but I can't seem to make sense of how to pass the file descriptor created by process substitution for a subsequent command in order to read what I just wrote.

Here's another example:

# can I make ./log a temporary file descriptor, an in-memory buffer?
exec 3>./log 4<./log
# writes
echo >&3 "hello"
# reads
cat <&4

Solution 1

If you want to globally redirect everything that happens to be written (like you do now), it's tricky, but can be hacked together.

I strongly recommend that, if it's possible, just do it by normal piping. just wrap everything you do in a subshell. In this case

(
 echo "this is the message"
 other stuff
) | cat

or just write everything into a variable with "$()" syntax.

The next way is to use what you did, but write to a tmpfs or /dev/shm if they are available. That's pretty straight forward, but you have to know what ram-based filesystems are in place (and set them up if possible).

Another way is to create a fifo with mkfifo. In both cases, you need to clean up after yourself.

EDIT:

I have a very ugly hack, but I bet someone can improve it.

#!/bin/bash
exec 3>&1
exec > >( tee >( ( tac && echo _ ) | tac | (read && cat > ./log) ) )

echo "lol"
sleep 5
echo "lol"

echo "finished writing"

exec >&-
exec >&3
exec 3>&-
echo "stdout now reopen"
sleep 1 #wait if the file is still being written asynchronously
cat ./log

How it works: first, you have a tee so you can see what's going on. This in turn outputs to another process substitution. There, you have the trick tac|tac which (because tac needs entire input to start outputting) waits for the entire stream to finish before going on. The last piece is in a subshell that actually outputs this into a file. Of course, the final shell would, immeadiately upon instantiation, create the output file in the filesystem if that was the only line. So something that also waits for the input to finally come, has to be done first, to delay file creation. I do this by outputting a dummy line first with echo, and then reading and discarding it. The read blocks until you close the file descriptor, signalling to tac its time has come. Hence, the closing of the stdout file descriptor at the end. I also saved the original stdout before opening the process substitution, in order to restore it at the end (to use cat once more). There's a sleep 5 in there, so I could check with ls if the file really wasn't created too early. The final sleep is trickier... The subshell is asynchronous and if there is a lot of output, you are waiting for both tacs to do their thing before the file is really there. So reasonably, you'll probably need to do something else to check if the thing really is finished. For instance, && touch sentinel at the end of the last subshell, and then while [ ! -f sentinel ]; do sleep 1; done && rm sentinel before you finally use the file.

All in all, two process substitutions and in the inner one another two subshells and 2 pipes. It's one of the ugliest things I've ever written... but it should create the file only when you close the stdout, which means it's well controlled and can be done when your filesystems are ready.

Solution 2

simple case:

output="$(echo "here is a message")"
# [...]
echo "$output"

Not so simple if you have to read from a redirection and cannot use a pipeline like

echo "$output" | while IFS= read -r line; do :; done

You can use a FIFO and a background process:

mkfifo fifo
echo "$output" >fifo &
while IFS= read -r line; do :; done <fifo

tee version

mkfifo fifo
( output="$(cat fifo)"; echo "$output" >fifo ) &
exec 3>&1
exec >fifo
...
exec 1>&3
cat fifo

Comments

[CMCDragonkai]

This doesn't solve my problem, because there's a reason why I'm redirecting all output to tee. It is because this bash snippet appears on top of a bunch of other bash commands that I don't control (don't want to modify). I can't change every subsequent command to output to a variable. And also I'm using tee because I still want to see interactive output.

[CMCDragonkai]

Unfortunately, I'm looking for a way so that I don't have to wrap all the bash command output.

[orion]

Then replacing ./log with a fifo or a temp file is a perfect choice. That's why in-memory filesystems exist.

[CMCDragonkai]

Can't use a temp file because the root filesystem gets changed in the middle of the script. This happens at the initrd level. If I save output to a temp file or fifo, I cannot read from it at the very end, because that file will be lost when the root filesystem gets swapped.

[orion]

@CMCDragonkai I've expanded my answer... I hope it's useful, or at least makes you laugh.

[Hauke Laging]

@CMCDragonkai I guess now I have what you need.

[CMCDragonkai]

Its pretty cool. Bash should really have the ability to create read/writable pseudofiles in the future though.

[orion]

Seems like a good idea at first, yes :) But it's a bit apart from the philosophy here. Unix-based systems go the other way (everything is a file, even things that aren't really), precisely because then, you can use all the available tools on everything transparently (no "hidden" stuff - apart from the posix message queue which is totally obscure). Bash would have to emulate a full IO interface - that can be done, it's called fuse, and I'm sure it exists. Just not within bash. And in most cases, shm/tmp works perfectly for this kind of thing (shm should work in your case too).

[CMCDragonkai]

I found out that there is a bash feature that allows what I want. It's called coproc. It allows to read and write to a subprocess. So you can create a pseudofile in way or an in-memory buffer.