Python- How to flush the log? (django)

Question

I'm working with Django-nonrel on Google App Engine, which forces me to use logging.debug() instead of print().

The "logging" module is provided by Django, but I'm having a rough time using it instead of print().

For example, if I need to verify the content held in the variable x, I will put
logging.debug('x is: %s' % x). But if the program crashes soon after (without flushing the stream), then it never gets printed.

So for debugging, I need debug() to be flushed before the program exits on error, and this is not happening.

Solution 1

I think this may work for you, assuming you're only using one(or default) handler:

>>> import logging
>>> logger = logging.getLogger()
>>> logging.debug('wat wat')
>>> logger.handlers[0].flush()

It's kind of frowned upon in the documentation, though.

Application code should not directly instantiate and use instances of Handler. Instead, the Handler class is a base class that defines the interface that all handlers should have and establishes some default behavior that child classes can use (or override). http://docs.python.org/2/howto/logging.html#handler-basic

And it could be a performance drain, but if you're really stuck, this may help with your debugging.

Solution 2

If the use case is that you have a python program that should flush its logs when exiting, use logging.shutdown().

From the python documentation:

logging.shutdown()

Informs the logging system to perform an orderly shutdown by flushing and closing all handlers. This should be called at application exit and no further use of the logging system should be made after this call.

Solution 3

I struggled with a similar problem and here is how I solved it. Instead of using logging module directly to output your logs, initialize your own logger as follows:

import sys
import logging


def init_logger():
    logger = logging.getLogger()

    h = logging.StreamHandler(sys.stdout)
    h.flush = sys.stdout.flush
    logger.addHandler(h)

    return logger

Then, use it instead of logging in your code:

def f():
    logger = init_logger()
    logger.debug('...')

As a result, you won't have problems with flushing logs anymore.

Solution 4

Django logging relies on the standard python logging module.

This module has a module-level method: logging.shutdown() which flushes all of the handlers and shuts down the logging system (i.e. logging can not longer be used after it is called)

Inspecting the code of this function shows that currently (python 2.7) the logging module holds a list of weak references to all handlers in a module-level variable called _handlerList so all of the handlers can be flushed by doing something like

[h_weak_ref().flush() for h_weak_ref in logging._handlerList]

because this solution uses the internals of the module @Mikes solution above is better, but it relies on having access to a logger, it can be generalized as follows:

 [h.flush() for h in my_logger.handlerList]

View more solutions

Comments

[Joseph Garvin]

Uh, how awre people supposed to flush their logs if the only way is through a handler that you're not supposed to access? That sounds broken.

[Hjulle]

No, that is not at all what the documentation says. The quote above means that you should only instantiate and use subclasses of Handler, instead of Handler directly. Your code only uses whatever handler classes are already instantiated by someone else, so there is no problem.

[GhostCat]

Very helpful indeed; I still find it extremely strange that I am not able to tell the stupid logger in its config that it always flush; but that my client code has to enforce that.

[Mario Stefanutti]

It returns "out of range". I initialized the logger using a .ini file with two handlers. logging.hasHandlers() returns true. logger.handlers[0].flush() IndexError: list index out of range

[idbrii]

What does h.flush = sys.stdout.flush do? You're replacing the flush implementation in the StreamHandler with the on from stdout? How does that help?

[Myles Prather]

This does not work (at least in Python 3.9). I don't know why it's upvoted.