How to redirect stderr in Python?

Question

I would like to log all the output of a Python script. I tried:

import sys

log = []

class writer(object):
    def write(self, data):
        log.append(data)

sys.stdout = writer()
sys.stderr = writer()

Now, if I "print 'something' " it gets logged. But if I make for instance some syntax error, say "print 'something# ", it wont get logged - it will go into the console instead.

How do I capture also the errors from Python interpreter?

I saw a possible solution here:

http://www.velocityreviews.com/forums/showpost.php?p=1868822&postcount=3

but the second example logs into /dev/null - this is not what I want. I would like to log it into a list like my example above or StringIO or such...

Also, preferably I don't want to create a subprocess (and read its stdout and stderr in separate thread).

Solution 1

You can't do anything in Python code that can capture errors during the compilation of that same code. How could it? If the compiler can't finish compiling the code, it won't run the code, so your redirection hasn't even taken effect yet.

That's where your (undesired) subprocess comes in. You can write Python code that redirects the stdout, then invokes the Python interpreter to compile some other piece of code.

Solution 2

I have a piece of software I wrote for work that captures stderr to a file like so:

import sys
sys.stderr = open('C:\\err.txt', 'w')

so it's definitely possible.

I believe your problem is that you are creating two instances of writer.

Maybe something more like:

import sys

class writer(object):
    log = []

    def write(self, data):
        self.log.append(data)

logger = writer()
sys.stdout = logger
sys.stderr = logger

Solution 3

I can't think of an easy way. The python process's standard error is living on a lower level than a python file object (C vs. python).

You could wrap the python script in a second python script and use subprocess.Popen. It's also possible you could pull some magic like this in a single script:

import os
import subprocess
import sys

cat = subprocess.Popen("/bin/cat", stdin=subprocess.PIPE, stdout=subprocess.PIPE)
os.close(sys.stderr.fileno())
os.dup2(cat.stdin.fileno(), sys.stderr.fileno())

And then use select.poll() to check cat.stdout regularly to find output.

Yes, that seems to work.

The problem I foresee is that most of the time, something printed to stderr by python indicates it's about to exit. The more usual way to handle this would be via exceptions.

---------Edit

Somehow I missed the os.pipe() function.

import os, sys
r, w = os.pipe()
os.close(sys.stderr.fileno())
os.dup2(w, sys.stderr.fileno())

Then read from r

Solution 4

For such a request, usually it would be much easier to do it in the OS instead of in Python.

For example, if you're going to run "a.py" and record all the messages it will generate into file "a.out", it would just be

python a.py 2>&1 > a.out

The first part 2>&1 redirects stderr to stdout (0: stdin, 1:stdout, 2:stderr), and the second redirects that to a file called a.out.

And as far as I know, this command works in Windows, Linux or MacOS! For other file redirection techniques, just search the os plus "file redirection"

Solution 5

To route the output and errors from Windows, you can use the following code outside of your Python file:

python a.py 1> a.out 2>&1

Source: https://support.microsoft.com/en-us/help/110930/redirecting-error-messages-from-command-prompt-stderr-stdout

View more solutions

Comments

[Hamish Grubijan]

Perhaps writer should implement writelines() as well. Do not forget sys.stderr.flush()

[Decula]

"print 'something#" shouldn't work as missing a single quotation mark. However, how do you check whether your print is being logged? You store stdout and stderr in a list. All your print results should go to the list. How do you print out the list content?

[EcirH]

You are right, of course it can't do it. Sorry for the stupid question. Another solution would be to "exec" the script inside another script.

[Admin]

Not all code is compiled at once. import statements are one example.

[Admin]

Another use case (though not EcirH's): capturing stderr from a C-library called from python.

[KingRadical]

In that case, just add a dummy flush method to your writer class: def flush(self): pass

[Ethan Bierlein]

Python code isn't compiled. It's interpreted on-the-fly.

[yantrab]

@EthanBierlein Python is indeed compiled, but to bytecode, not machine instructions. The point is that there is a compilation step, which is what produces SyntaxErrors. If a SyntaxError occurs, then the code will not run, so there is nothing you can do in the code to log syntax errors.

[Decula]

user file handler as log handler, neat. How about multiple modules access the log list/file at the same time?

[Joe M.]

And for those who want to get the output as a string: ` with io.StringIO() as buff: with contextlib.redirect_stderr(buff): help(pow) print(buff.getvalue()) `

[Guimoute]

@Decula I have no problem with that actually. My main programs have a redirection of sys.stdout to a pyQt5 widget that can display text, and all print(...) from modules are redirected too without having to incorporate sys.stdout = something_else.

[CrippledTable]

It is possible, the answer is below: stackoverflow.com/a/1956228/8304478