How to capture terminal output as execution proceeds?

Question

I have tried using all the commands mention in Byte Commander's answer to the question here but they do not work unless the program finishes.

I am running a python script like so 'python script.py' and have tried replacing 'command' in Byte Commander's answer with 'python script.py' however the terminal output is not shown in the terminal anymore with any of the commands and only gets written to the file output.txt in the case that the python script completes (actually I've discovered it comes in large chunks as it proceeds, but not line-by-line as I require). I believe it may be because the python script calls another non-python program in another shell (it calls a finite element package called gmesh). The code takes a long time (several hours) to complete and I want to be able to see the output written to the file as it proceeds so I can see it's progress and what the terminal output looks like even if the program crashes part way through. How might I be able to go about this?

I've added a video to demonstrate the issue. When the program runs without capture the information comes line by line. When I attempt to capture the output it comes in chunks, when these chunks takes hours and it crashes during this process I get no information on where it crashed.

Solution

You can do as in the following example.

Suppose we have a script called myscript.py that looks like this:

import time
for x in range(0, 33):
   print("hello", x, flush=True)
   time.sleep(1)

Then if we run it like this:

python3 myscript.py > mylog.txt

it will hang there until it completes, so we will not see the output while it is running.

To be able to see the output while it is running, we can do this instead:

python3 myscript.py > mylog.txt &

where the ampersand means that the terminal will not hang, we can give more commands while it is running. Then, we can do this to look at the contents of the log file while myscript.sh is writing to it:

tail -f mylog.txt

(Another possibility is to open a separate terminal window and do tail -f mylog.txt from there.)

Note that for this to work, the flush=True part of the print line is important, without flush=True the file contents will only be seen after the program completes.

In case the output is generated from some other program inside your python script, you can try adding import sys and then doing sys.stdout.flush() in the python code after calling the other program. See https://stackoverflow.com/a/230774/6708867

Comments

[meuh]

check out the other answer. You need to run unbuffer gmesh (or stdbuf -o0 gmesh if you have that cmd).

[Sergiy Kolodyazhnyy]

It's somewhat unclear. So you want to both see output on screen AND save to file at the same time (probably to review later)?

[tokosh]

I'm not sure if I understand. Would tee not be sufficient? Example: ./executable | tee log.txt

[SomeRandomPhysicist]

This doesn't work for my program. I've tried all of the commands in the other answer, including the one you use above. I have been looking at the file with another seperate terminal window using watch -n 0.5 "tail output.txt" to constantly monitor the file and output is only written when certain functions complete. I've been running a script that only takes a few 10s of seconds instead of the multiple hour script. I think it writes output to the file when the shell it creates is closed.

[SomeRandomPhysicist]

I believe the shell spawns another shell to run the program and that output is not captured by any of these commands until the spawned shell terminates. However if I run the python script without capturing the output it displays the output as the program progresses so I can see the progress.

[Elias]

@SomeRandomPhysicist maybe you need to add "flush" for the output you are doing inside the python program. I edited my answer above to show an example of that. Does that help?

[SomeRandomPhysicist]

The issue is that the output comes from a program called gmesh, which is called by a python package called pygmesh which I am using. I've also added a video to demonstrate the issue I am having and the behaviour I am seeing.

[Elias]

@SomeRandomPhysicist how about adding sys.stdout.flush() then (I edited the answer again to include that idea)

[SomeRandomPhysicist]

I agree this would work if I was generating the text output myself within python but I am using the following package pypi.org/project/pygmsh which itself calls another program gmsh.info which generates the output I am trying to capture.