Write Python stdout to file immediately
Solution 1
This is happening because normally when process STDOUT is redirected to something other than a terminal, then the output is buffered into some OS-specific-sized buffer (perhaps 4k or 8k in many cases). Conversely, when outputting to a terminal, STDOUT will be line-buffered or not buffered at all, so you'll see output after each \n
or for each character.
You can generally change the STDOUT buffering with the stdbuf
utility:
stdbuf -oL python script.py > log
Now if you tail -F log
, you should see each line output immediately as it is generated.
Alternatively explicit flushing of the output stream after each print should achieve the same. It looks like sys.stdout.flush()
should achieve this in Python. If you are using Python 3.3 or newer, the print
function also has a flush
keyword that does this: print('hello', flush=True)
.
Solution 2
This should do the job:
import time, sys
for i in range(10):
print('bla')
sys.stdout.flush()
time.sleep(5)
As Python will buffer the stdout
by default, here i have used sys.stdout.flush()
to flush the buffer.
Another solution would be to use the -u
(unbuffered) switch of python
. So, the following will do too:
python -u script.py >> log
Solution 3
Variation on the theme of using python's own option for unbuffered output would be to use #!/usr/bin/python -u
as first line.
With #!/usr/bin/env python
that extra argument not gonna work, so alternatively,one could run PYTHONUNBUFFERED=1 ./my_scriipt.py > output.txt
or do it in two steps:
$ export PYTHONUNBUFFERED=1
$ ./myscript.py
Solution 4
You should pass flush=True
to the print
function:
import time
for i in range(10):
print('bla', flush=True)
time.sleep(5)
According to the documentation, by default, print
doesn't enforce anything about flushing:
Whether output is buffered is usually determined by file, but if the
flush
keyword argument is true, the stream is forcibly flushed.
And the documentation for sys
's strems says:
When interactive, standard streams are line-buffered. Otherwise, they are block-buffered like regular text files. You can override this value with the
-u
command-line option.
If you are stuck with an ancient version of python you have to call the flush
method of the sys.stdout
stream:
import sys
import time
for i in range(10):
print('bla')
sys.stdout.flush()
time.sleep(5)
Related videos on Youtube
Bart
I'm a researcher at Wageningen University, working with Python, C++, CUDA-C (and Fortran when forced to) on numerical modeling of (turbulence in) the atmosphere, partially with models that I co-developed like CLASS (mixed-layer model) or MicroHH (DNS/LES model).
Updated on September 18, 2022Comments
-
Bart over 1 year
When trying to write the stdout from a Python script to a text file (
python script.py > log
), the text file is created when the command is started, but the actual content isn't written until the Python script finishes. For example:script.py:
import time for i in range(10): print('bla') time.sleep(5)
prints to stdout every 5 seconds when called with
python script.py
, but when I callpython script.py > log
, the size of the log file stays zero until the script finishes. Is it possible to directly write to the log file, such that you can follow the progress of the script (e.g. usingtail
)?EDIT It turns out that
python -u script.py
does the trick, I didn't know about the buffering of stdout.-
Admin almost 5 years@jezmck, I could have understood question wrong.
-
-
Bart over 9 yearsThanks, I didn't know about the buffering! Knowing that, Google pretty quickly told me that
python -u script.py
does the trick. EDIT So many answers at once, I accepted yours since it pointed me in the direction of the buffering. -
Digital Trauma over 9 years@julbra Cool, yes I didn't know python had that option either. Some command-line programs also have similar options - e.g.
--line-buffered
forgrep
, but some others don't.stdbuf
is the general catchall utility to deal with those that don't. -
heemayl over 9 years@DigitalTrauma: Isn't it better to use no buffering at all i.e.
stdbuf -o0 python script.py > log
in this kind of determined circumstances? -
Digital Trauma over 9 years@heemayl
-oL
is a compromise. In general larger buffers will provide better performance when redirecting somewhere (fewer system calls and fewer I/O operations). However if it is absolutely necessary to see each character as it is output then yes,-o0
would be required. -
Bart over 9 yearsThe flush=True argument works nicely with Python 3.4.2, indeed doesn't work with the ancient (..) Python 2.7.9
-
dotancohen over 9 yearsThis answer suggests the same thing that
DigitalTrauma
said 10 hours prior. You should upvote his post, not post the same thing again. -
Bakuriu over 9 years@Paul Please avoid copy pasting contents between answers, or at the bery least mention the original authors that provided the content.
-
dotancohen over 9 years@Bakuriu: I'm sorry then! This shows a good reason to always post why when downvoting. Could you please edit the post a bit so that I can change my downvote to an upvote? Thank you!
-
Paul over 9 years@Bakuriu I wrote the edit myself. What answer do you think I copy-pasted the text from? Edit 1: I see you mentioned the same thing in your own answer. Again, I didn't copy anything. In fact, DigitalTrauma's answer was the only one I read when I edited it. Edit 2: And to be fair, you didn't mention anything about this being available only since Python 3.3.
-
Bakuriu over 9 years@Paul Before introducing new content in an answer you should look the other answers, to avoid these kind of situations. Only edits about formatting/adding references and such don't need the context from other answers. (Btw: Python<3.3 is my definition of ancient version of python.)
-
CuriousMarc about 4 yearsOh my, many thanks! I was pulling my hair out for hours: why would I not see any output from a simple pipe from a python program streaming output piped to sed? Worked in Ubuntu, but not on Debian, at least not until I killed the python program. Turns out the output was just caught in the stdout buffer on Debian! But not on Ubuntu. Go figure! Your stdbuf -oL solution fixed it instantly.