How to make output of any shell command unbuffered?

Question

Is there a way to run shell commands without output buffering?

For example, hexdump file | ./my_script will only pass input from hexdump to my_script in buffered chunks, not line by line.

Actually I want to know a general solution how to make any command unbuffered?

Solution 1

AFAIK, you can't do it without ugly hacks. Writing to a pipe (or reading from it) automatically turns on full buffering and there is nothing you can do about it :-(. "Line buffering" (which is what you want) is only used when reading/writing a terminal. The ugly hacks exactly do this: They connect a program to a pseudo-terminal, so that the other tools in the pipe read/write from that terminal in line buffering mode. The whole problem is described here:

• http://www.pixelbeat.org/programming/stdio_buffering/

The page has also some suggestions (the aforementioned "ugly hacks") what to do, i.e. using unbuffer or pulling some tricks with LD_PRELOAD.

Solution 2

Try stdbuf, included in GNU coreutils and thus virtually any Linux distro. This sets the buffer length for input, output and error to zero:

stdbuf -i0 -o0 -e0 command

Solution 3

The command unbuffer from the expect package disables the output buffering:
Ubuntu Manpage: unbuffer - unbuffer output

Example usage:

unbuffer hexdump file | ./my_script

Solution 4

You could also use the script command to make the output of hexdump line-buffered (hexdump will be run in a pseudo terminal which tricks hexdump into thinking its writing its stdout to a terminal, and not to a pipe).

# cf. http://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe/
stty -echo -onlcr
script -q /dev/null hexdump file | ./my_script         # FreeBSD, Mac OS X
script -q -c "hexdump file" /dev/null | ./my_script    # Linux
stty echo onlcr

View more solutions

Comments

[Trevor Boyd Smith]

more upvoted question unix.stackexchange.com/questions/25372

[bodacydo]

Thanks. I had never heard about pseudo terminals before.

[Gringo Suave]

I used the -f parameter to flush, not sure if necessary but it worked.

[ElDog]

This worked better than unbuffer for me. stdbuf passed any signals (SIGUSR2 in my case) I sent to it to the command (which is what I wanted to happen), while unbuffer didn't seem to want to.

[olejorgenb]

unbuffer seems to merge stdout and stderr of the command though... (!)

[TerrenceSun]

In my case, stdbuf failed, but script works like a charm. The program buffers the output when it is being piped to tee, but not when running in terminal. So, script works! Thanks!

[Nordic Mainframe]

stdbuf uses one the aforementioned LD_PRELOAD tricks to do this and hence does not work with statically linked or setuid executables. See this question for a discussion: stackoverflow.com/questions/13644024/…

[xmnboy]

Very clever solution, and it works!! Thank you. :smile:

[AFH]

I found this very useful, but limited, as it does not run scripts implicitly. However, stdbuf -o0 bash runs a complete session with all my scripts and aliases available, and with no output buffering on any of the commands, which is exactly what I wanted. Buffering was of course restored on exit from this bash instance. I'm using Ubuntu 16.04.

[RandomDSdevel]

Only one problem with that: per script's man page as referenced by this, "Certain interactive commands, such as vi(1), create garbage in the typescript file. Script works best with commands that do not manipulate the screen, the results are meant to emulate a hardcopy terminal."

[toolforger]

@olejorgenb yes it merges stdout and stderr, that's by design: unbuffer uses expect, and expect is there to emulate the human in human-program interaction, and humans don't get to see whether an output is stdout or stderr.

[IcarusNM]

I liked this so much, I put this first in my .bash_profile. if [[ "$0" == "-bash" ]]; then exec /usr/bin/stdbuf -i0 -oL -eL /bin/bash -l; fi

[JinnKo]

Worked for me in an exec pattern for sending output to a logfile, with all output line buffered rather than buffering fully disabled: exec > >(stdbuf -i0 -oL -eL awk '{print strftime("%Y-%m-%d %H:%M:%S"), $0 }' | stdbuf -i0 -oL -eL tee "$LOGFILE") 2>&1