How to redirect stdout and stderr to logger in Python

Question

I have a logger that has a RotatingFileHandler. I want to redirect all Stdout and Stderr to the logger. How to do so?

Solution 1

Not enough rep to comment, but I wanted to add the version of this that worked for me in case others are in a similar situation.

class LoggerWriter:
    def __init__(self, level):
        # self.level is really like using log.debug(message)
        # at least in my case
        self.level = level

    def write(self, message):
        # if statement reduces the amount of newlines that are
        # printed to the logger
        if message != '\n':
            self.level(message)

    def flush(self):
        # create a flush method so things can be flushed when
        # the system wants to. Not sure if simply 'printing'
        # sys.stderr is the correct way to do it, but it seemed
        # to work properly for me.
        self.level(sys.stderr)

and this would look something like:

log = logging.getLogger('foobar')
sys.stdout = LoggerWriter(log.debug)
sys.stderr = LoggerWriter(log.warning)

Solution 2

UPDATE for Python 3:

• Including a dummy flush function which prevents an error where the function is expected (Python 2 was fine with just linebuf='').
• Note that your output (and log level) appears different if it is logged from an interpreter session vs being run from a file. Running from a file produces the expected behavior (and output featured below).
• We still eliminate extra newlines which other solutions do not.

class StreamToLogger(object):
    """
    Fake file-like stream object that redirects writes to a logger instance.
    """
    def __init__(self, logger, level):
       self.logger = logger
       self.level = level
       self.linebuf = ''

    def write(self, buf):
       for line in buf.rstrip().splitlines():
          self.logger.log(self.level, line.rstrip())

    def flush(self):
        pass

Then test with something like:

import StreamToLogger
import sys
import logging

logging.basicConfig(
        level=logging.DEBUG,
        format='%(asctime)s:%(levelname)s:%(name)s:%(message)s',
        filename='out.log',
        filemode='a'
        )
log = logging.getLogger('foobar')
sys.stdout = StreamToLogger(log,logging.INFO)
sys.stderr = StreamToLogger(log,logging.ERROR)
print('Test to standard out')
raise Exception('Test to standard error')

See below for old Python 2.x answer and the example output:

All of the prior answers seem to have problems adding extra newlines where they aren't needed. The solution that works best for me is from http://www.electricmonk.nl/log/2011/08/14/redirect-stdout-and-stderr-to-a-logger-in-python/, where he demonstrates how send both stdout and stderr to the logger:

import logging
import sys
 
class StreamToLogger(object):
   """
   Fake file-like stream object that redirects writes to a logger instance.
   """
   def __init__(self, logger, log_level=logging.INFO):
      self.logger = logger
      self.log_level = log_level
      self.linebuf = ''
 
   def write(self, buf):
      for line in buf.rstrip().splitlines():
         self.logger.log(self.log_level, line.rstrip())
 
logging.basicConfig(
   level=logging.DEBUG,
   format='%(asctime)s:%(levelname)s:%(name)s:%(message)s',
   filename="out.log",
   filemode='a'
)
 
stdout_logger = logging.getLogger('STDOUT')
sl = StreamToLogger(stdout_logger, logging.INFO)
sys.stdout = sl
 
stderr_logger = logging.getLogger('STDERR')
sl = StreamToLogger(stderr_logger, logging.ERROR)
sys.stderr = sl
 
print "Test to standard out"
raise Exception('Test to standard error')

The output looks like:

2011-08-14 14:46:20,573:INFO:STDOUT:Test to standard out
2011-08-14 14:46:20,573:ERROR:STDERR:Traceback (most recent call last):
2011-08-14 14:46:20,574:ERROR:STDERR:  File "redirect.py", line 33, in 
2011-08-14 14:46:20,574:ERROR:STDERR:raise Exception('Test to standard error')
2011-08-14 14:46:20,574:ERROR:STDERR:Exception
2011-08-14 14:46:20,574:ERROR:STDERR::
2011-08-14 14:46:20,574:ERROR:STDERR:Test to standard error

Note that self.linebuf = '' is where the flush is being handled, rather than implementing a flush function.

Solution 3

If it's an all-Python system (i.e. no C libraries writing to fds directly, as Ignacio Vazquez-Abrams asked about) then you might be able to use an approach as suggested here:

class LoggerWriter:
    def __init__(self, logger, level):
        self.logger = logger
        self.level = level

    def write(self, message):
        if message != '\n':
            self.logger.log(self.level, message)

and then set sys.stdout and sys.stderr to LoggerWriter instances.

Solution 4

You can use redirect_stdout context manager:

import logging
from contextlib import redirect_stdout

logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
logging.write = lambda msg: logging.info(msg) if msg != '\n' else None

with redirect_stdout(logging):
    print('Test')

or like this

import logging
from contextlib import redirect_stdout


logger = logging.getLogger('Meow')
logger.setLevel(logging.INFO)
formatter = logging.Formatter(
    fmt='[{name}] {asctime} {levelname}: {message}',
    datefmt='%m/%d/%Y %H:%M:%S',
    style='{'
)
ch = logging.StreamHandler()
ch.setLevel(logging.INFO)
ch.setFormatter(formatter)
logger.addHandler(ch)

logger.write = lambda msg: logger.info(msg) if msg != '\n' else None

with redirect_stdout(logger):
    print('Test')

Solution 5

As an evolution to Cameron Gagnon's response, I've improved the LoggerWriterclass to:

class LoggerWriter(object):
    def __init__(self, writer):
        self._writer = writer
        self._msg = ''

    def write(self, message):
        self._msg = self._msg + message
        while '\n' in self._msg:
            pos = self._msg.find('\n')
            self._writer(self._msg[:pos])
            self._msg = self._msg[pos+1:]

    def flush(self):
        if self._msg != '':
            self._writer(self._msg)
            self._msg = ''

now uncontrolled exceptions look nicer:

2018-07-31 13:20:37,482 - ERROR - Traceback (most recent call last):
2018-07-31 13:20:37,483 - ERROR -   File "mf32.py", line 317, in <module>
2018-07-31 13:20:37,485 - ERROR -     main()
2018-07-31 13:20:37,486 - ERROR -   File "mf32.py", line 289, in main
2018-07-31 13:20:37,488 - ERROR -     int('')
2018-07-31 13:20:37,489 - ERROR - ValueError: invalid literal for int() with base 10: ''

View more solutions

Comments

[orenma]

thank you, that did the job, but for some reason stderr send it's message each word separately, do you know why?

[Vinay Sajip]

@orenma presumably because write is called word-by-word. You can adapt my example code to suit your needs more closely.

[Moberg]

What if sys.stderr.flush() is called after redirecting stderr?

[Vinay Sajip]

@Moberg then you have to define a flush() method, too.

[Moberg]

I cant make library code not use sys.stderr .flush() etc. What is the best way to handle all its attributes?

[asmeurer]

This code is licensed GPL. I'm not sure if it can even be posted on SO, which requires compatibility with CC by-sa.

[azmath]

What if C libraries are involved? Then what? How to get the C library to output to the same LoggerWriter?

[pawamoy]

I get a weird output because of the flush method: warning archan_pylint:18: <archan_pylint.LoggerWriter object at 0x7fde3cfa2208>. It seems the stderr object is printed rather than a newline or else, so I just removed the flush method and it seems to work now.

[soungalo]

Any idea why I'm getting this error message? "Exception ignored in: <__main__.StreamToLogger object at 0x7f72a6fbe940> AttributeError 'StreamToLogger' object has no attribute 'flush' "

[Toni Homedes i Saun]

@Cameron Please look at my answer below for a small improvement in output readability.

[Attila123]

It works both with python2 and 3, in case you log to a file (e.g. logging.basicConfig(filename='example.log', level=logging.DEBUG). But if you want e.g. logging.basicConfig(stream=sys.stdout, level=logging.DEBUG) then it does not work (on python3 it also causes stack overflow). (I guess because it captures std out), so not so useful for logging e.g. from a Kubernetes pod to std out. Note that the code found by shellcat_zero does work also with stream=sys.stdout.

[Attila123]

docs.python.org/3/library/contextlib.html : "Context manager for temporarily redirecting sys.stdout to another file or file-like object." "Note that the global side effect on sys.stdout means that this context manager is not suitable for use in library code and most threaded applications. It also has no effect on the output of subprocesses. However, it is still a useful approach for many utility scripts." So it seems quite inconvenient (if possible at all) to cover a whole application (e.g. I have a microservice which runs a grpc server, which starts threads when serving requests).

[OmerB]

Note that using an empty flush() method as done here is ok, since the logging handler handles flushing internally: stackoverflow.com/a/16634444/8425408

[Preetkaran Singh]

Remove the last two lines of the code snippet, the error message goes away....

[Alon Gouldman]

def flush(self): pass avoids printing <archan_pylint.LoggerWriter object at 0x7fde3cfa2208> to the log

[James H]

It's 'safer' to extend TextIOBase. Somewhere in my library is calling sys.stdout.isatty() and the StreamToLogger failed because of no attribute 'isatty'. It works after I define class StreamToLogger(TextIOBase).

[xjcl]

Don't the other solutions here have the same global side effect on sys.stdout/sys.stderr as well? @Attila123

[xjcl]

You are right, the top answers produce spurious newlines on e.g. Exceptions. My answer follows a very similar approach.

[Toni Homedes i Saun]

Nice but you are assuming that if there is a '\n' in msg it will always be at the end of msg, and that a single msg will never have more than one '\n'. This is probably true in most current Python implementations, but I'm not sure if it is defined as a language standard, so I prefer the "check each time" approach. It is not as bad as it seems because each time Shlemiel gets a new paint bucket (a '\n') he brings it to the current painting point, starting anew from zero.

[xjcl]

@ToniHomedesiSaun A message with '\n's in it is ok and will just be printed as a multiline log, but as you said most internal error messages are chunked calls to sys.stderr and will appear as separate logs. But I guess you could also msg.split('\n') if you are not ok with multiline logs.