How to make shell output redirect (>) write while script is still running?

python linux bash shell command-line

18,663

Solution 1

You need to flush the output sys.stdout.flush() (or smth) if you want to see it immediately. See this

Solution 2

stdout is buffered, if not connected to terminal.

You can change this policy to line-buffering via stdbuf

stdbuf -oL python script.py > ~/public_html/results.txt

So you don't have to flush in your Python script and keep it IO efficient, if line-buffering is not required.

Solution 3

I suspect the file is being continuously written, but that the web server is reporting the modified date of the file as the time it was opened, and thus is reporting that no change to the file has occurred and the result is being cached (either at the web server or at the client).

I would first try a forced reload (Ctrl+F5 or Ctrl+Shift+R or Shift+<reload_button>) and see if that helps. If it doesn't, then you can try something else.

In a separate shell on the server, do

tail -f ~/public_html/results.txt

Tail prints out the last n lines of the file (where n defaults to 10), but the -f parameter monitors the file and continues to report output as the file grows. This will at least give you confidence that the file is being written to incrementally.

I hope that helps.

18,663

Author by

noio

Game designer & developer.

Updated on June 13, 2022

Comments

noio almost 2 years
I wrote a short script that never terminates. This script continuously generates output that I have to check on every now and then. I'm running it on a lab computer through SSH, and redirecting the output to a file in my public_html folder on that machine.
```
python script.py > ~/public_html/results.txt
```
However, the results don't show up immediately when I refresh the address. The results show up when I terminate the program, but as I said, it doesn't halt by itself. Is that redirect (>) being lazy with with writing? Is there a way to continuously (or with an interval) update the results in the file?

Or is it the webserver that doesn't update the file while it is still being written?
noio almost 14 years

Oh thanks, I'll try it right away. I didn't think it was python buffering output. --edit: I tried, it makes no difference. so either > is buffering, or the webserver.
noio almost 14 years

Nevermind the previous comment, I was being impatient. It' still not "real-time" but that's probably browser cache.
noio almost 14 years

Thanks, nc3b's solution worked. Tail shows the newest results as they are written. The browser view updates in batches every short period of time now. Forced refresh doesn't change that. But it's quick enough.
noio almost 14 years

Will this be less efficient than python's flush() if called sparingly? I use flush() now and it's called less than once per line.
Jürgen Hötzel almost 14 years

I doubt there will be a difference, because data will most likely not be written to disk if you don't use flush(3). If you use your app on a terminal line-buffering will be default. For long running background jobs, redirecting to a file will choose more efficient buffering. For special cases like yours, you can always use stdbuf and thus keep IO-Code out of your app, keeping it flexible and more readable.
simao about 13 years

Where can I find stdbuf on ubuntu? doesn't seem to be included in the coreutils package
Eric Cousineau about 6 years

I actually prefer this over the accepted answer, as it's more general and does not require source modification. (Or in more technical terms, it's absolutely ballin'.)