Temporarily Redirect stdout/stderr

Question

Is it possible to temporarily redirect stdout/stderr in Python (i.e. for the duration of a method)?

Edit:

The problem with the current solutions (which I at first remembered but then forgot) is that they don't redirect; rather, they just replace the streams in their entirety. Hence, if a method has a local copy of one the variable for any reason (e.g. because the stream was passed as a parameter to something), it won't work.

Any solutions?

Solution 1

To solve the issue that some function might have cached sys.stdout stream as a local variable and therefore replacing the global sys.stdout won't work inside that function, you could redirect at a file descriptor level (sys.stdout.fileno()) e.g.:

from __future__ import print_function
import os
import sys

def some_function_with_cached_sys_stdout(stdout=sys.stdout):
    print('cached stdout', file=stdout)

with stdout_redirected(to=os.devnull), merged_stderr_stdout():
    print('stdout goes to devnull')
    some_function_with_cached_sys_stdout()
    print('stderr also goes to stdout that goes to devnull', file=sys.stderr)
print('stdout is back')
some_function_with_cached_sys_stdout()
print('stderr is back', file=sys.stderr)

stdout_redirected() redirects all output for sys.stdout.fileno() to a given filename, file object, or file descriptor (os.devnull in the example).

stdout_redirected() and merged_stderr_stdout() are defined here.

Solution 2

You can also put the redirection logic in a contextmanager.

import os
import sys

class RedirectStdStreams(object):
    def __init__(self, stdout=None, stderr=None):
        self._stdout = stdout or sys.stdout
        self._stderr = stderr or sys.stderr

    def __enter__(self):
        self.old_stdout, self.old_stderr = sys.stdout, sys.stderr
        self.old_stdout.flush(); self.old_stderr.flush()
        sys.stdout, sys.stderr = self._stdout, self._stderr

    def __exit__(self, exc_type, exc_value, traceback):
        self._stdout.flush(); self._stderr.flush()
        sys.stdout = self.old_stdout
        sys.stderr = self.old_stderr

if __name__ == '__main__':

    devnull = open(os.devnull, 'w')
    print('Fubar')

    with RedirectStdStreams(stdout=devnull, stderr=devnull):
        print("You'll never see me")

    print("I'm back!")

Solution 3

starting from python 3.4 there is the context manager contextlib.redirect_stdout:

from contextlib import redirect_stdout

with open('yourfile.txt', 'w') as f:
    with redirect_stdout(f):
        # do stuff...

to completely silence stdout this works:

from contextlib import redirect_stdout

with redirect_stdout(None):
    # do stuff...

Solution 4

I am not sure what temporary redirection means. But, you can reassign streams like this and reset it back.

temp = sys.stdout
sys.stdout = sys.stderr
sys.stderr = temp

Also to write to sys.stderr within print stmts like this.

 print >> sys.stderr, "Error in atexit._run_exitfuncs:"

Regular print will to stdout.

Solution 5

It's possible with a decorator such as the following:

import sys

def redirect_stderr_stdout(stderr=sys.stderr, stdout=sys.stdout):
    def wrap(f):
        def newf(*args, **kwargs):
            old_stderr, old_stdout = sys.stderr, sys.stdout
            sys.stderr = stderr
            sys.stdout = stdout
            try:
                return f(*args, **kwargs)
            finally:
                sys.stderr, sys.stdout = old_stderr, old_stdout

        return newf
    return wrap

Use as:

@redirect_stderr_stdout(some_logging_stream, the_console):
def fun(...):
    # whatever

or, if you don't want to modify the source for fun, call it directly as

redirect_stderr_stdout(some_logging_stream, the_console)(fun)

But note that this is not thread-safe.

View more solutions

Comments

[David Wolever]

Redirecting stdout/stderr isn't uncommon (or, at least, not unheard of) — the answers here explain the process nicely.

[Fred Foo]

@TokenMacGuy: while you should never write library code that has outputs to stderr or stdout wired-in, you can't always avoid using that kind of code.

[Rob Cowie]

@Mehrdad Try replacing sys.__stdout__ first thing in your code, before you import the third-party modules

[user541686]

@Rob: I think you missed the 2nd word in my title. :-)

[Rob Cowie]

@Mehrdad Nope. Replace sys.__stdout__ early in your code with your own stream-like object (i.e. implements .write()). All references to sys.stdout point to it. Have it proxy to a changeable stream, defaulting to stdout. You should then have the ability to switch the proxied stream at will. I haven't tried this; I'm thinking out loud.

[user541686]

@Rob: Oh I see... it's an good idea (though overkill!); I might try it. :)

[jfs]

related: Redirect stdout to a file in Python?

[alexia]

I realize this is an old question, but for reference, you don't need to store the values of sys.stdout and sys.stderr. There is sys.__stdout__ and sys.__stderr__. (reference)

[user541686]

I guess I could wrap a try-finally around that; seems like it'd work, though not as pretty as I wanted. Thanks +1

[user541686]

+1 seems to be what I need. Just curious, why'd you do an except: raise?

[Fred Foo]

@Mehrdad: forget about the except: raise thing, it was a thinko on my part. I rewrote the whole thing btw.; I always get the level of nesting in decorators wrong the first time, but it actually works now.

[user541686]

You know, I noticed a problem: What if a program has cached the stream objects (e.g. passed them as a parameter to a function that accepts a stream)? Then this wouldn't work. :\ Is there a way to redirect the streams themselves, rather than replacing them with new streams?

[Fred Foo]

@Mehrdad: then you'd do some low-level file descriptor magic, or even fork the process and redirect the streams in the child before performing the desired code, returning the result across a pipe. There's no silver bullet here, I'm afraid.

[ThePracticalOne]

changereturn wrap(f) to return wrap

[JeffG]

This example works for those of us stuck on python 2.2. Thanks

[joedborg]

you don't need temp. sys.__stderr__ stays as the correct object.

[Gwen]

+1 for the cleanest and simplest solution I've seen to this problem

[Bakuriu]

-1 -- The with will not restore the initial sys.stdout. If you try to print something after those with blocks you receive a ValueError: I/O operation on closed file.. You should remove the second with and put a sys.stdout = sys.__stdout__ at the end of the first with block.

[jfs]

This answer replaces sys.stdout, sys.stderr instead of redirecting them as OP asked in the edited version of the question. See my answer that does redirect them.

[kevinarpe]

@RobCowie: Style question: Is there a reason why you make self._stdout "protected" (in the Python sense), vs self.__stdout "private" (in the Python sense)?

[Rob Cowie]

I don't seem to ever create private double-underscore properties. I've yet to find a situation that warrants it.

[dwanderson]

@kevinarpe - there's no difference between x._bar and x.__bar except that x.__bar is harder to get at, debugging wise due to name-mangling. It doesn't actually make the variable private, so it doesn't buy you anything, but makes your debugging life more difficult. In general, stick with a single underscore unless you're dealing with multiple inheritance and have a particular reason to want dunderscore (both classes' variables need to exist? I feel like there are more elegant solutions)

[twasbrillig]

In Python you don't need the temp variable. You can call sys.stdout, sys.stderr = sys.stderr, sys.stdout.

[Peter]

I just tried this with redirect_stdout(None) and it didn't work. The module producing the output is written in C++, maybe that has something to do with it.

[hiro protagonist]

how 'does it not work'? because it really should! see e.g. this question: stackoverflow.com/questions/49757674/…

[Peter]

Maybe it's just a corner case, I tried wrapping a call to caffe.Net() in with redirect_stderr(None) and still got a load of debug messaged dumped to stderr. Turns out you can disable those for the specific case of Caffe another way but I guess disabling standard output is not that easy for modules that use std::iostream under the hood.

[Pablo]

Is it possible to no redirect the stdout to a file rather to just ignore it ?

[Mustafa Aydın]

perhaps that with is better as with open(os.devnull, "w") as devnull, RedirectStdStreams(stdout=devnull, stderr=devnull): so that devnull is also closed fine

[baxx]

what if one wants to redirect both stdout and stderr to the same place?

[hiro protagonist]

@baxx with redirect_stdout(sys.stderr): and then you could redirect stderr (or the other way round).