How to pickle loggers?

Question

Working on a project that requires that I am able to pickle the container object at any point, since we expect it to fail on external conditions quite frequently and be able to fully pick up where we left off.

I'm using the python logging library quite extensively, and all of my classes start by setting up a logger like:

class foo:
   def __init__(self):
       self.logger = logging.getLogger("package.foo")

Since I'm pickling a container class, it has several layers of classes within it, each with their own logger instance.

Now, for some reason, these loggers are breaking Pickle. I'm getting the following error, which goes away if I delete self.logger from all the classes:

Can't pickle 'lock' object: <thread.lock object at ... >

So my question is whether or not there is some way to remove the lock objects from all the loggers without having to recurse through my whole object tree deleting loggers which I will have to recreate on unpickle.

Solution 1

Found a very similar question here, with an answer that worked for me:

How to stop attributes from being pickled in Python

edit: used this answer: How to stop attributes from being pickled in Python

Solution 2

You could also create a class that implements a property which returns the needed logger. Every class which inherits from this "LoggerMixin" is now able to use the logger in the same way you were using it before.

class LoggerMixin():
    @property
    def logger(self):
        component = "{}.{}".format(type(self).__module__, type(self).__name__)
        return logging.getLogger(component)

class Foo(LoggerMixin):
    def __init__(self):
        self.logger.info("initialize class")

    def bar(self):
        self.logger.info("execute bar")

Solution 3

You could create a class that wraps the logger and implements __getstate__ and __setstate__.

This is pasted from http://docs.python.org/library/pickle.html. The fh is handled in a way which may be similar to what you need.

#!/usr/local/bin/python

class TextReader:
    """Print and number lines in a text file."""
    def __init__(self, file):
        self.file = file
        self.fh = open(file)
        self.lineno = 0

    def readline(self):
        self.lineno = self.lineno + 1
        line = self.fh.readline()
        if not line:
            return None
        if line.endswith("\n"):
            line = line[:-1]
        return "%d: %s" % (self.lineno, line)

    def __getstate__(self):
        odict = self.__dict__.copy() # copy the dict since we change it
        del odict['fh']              # remove filehandle entry
        return odict

    def __setstate__(self, dict):
        fh = open(dict['file'])      # reopen file
        count = dict['lineno']       # read from file...
        while count:                 # until line count is restored
            fh.readline()
            count = count - 1
        self.__dict__.update(dict)   # update attributes
        self.fh = fh                 # save the file object

Solution 4

New in Python 3.7 (bpo30520)

Logger can now be pickled like many other objects.

import pickle
import logging
log = logging.getLogger(__name__)
logger_pickle = pickle.dumps(log)

# and of coarse, to load:
log = pickle.loads(logger_pickle)

Solution 5

You could have used dill here, which can pickle loggers and locks.

>>> class foo:
...    def __init__(self):
...        self.logger = logging.getLogger("package.foo")
... 
>>> import dill
>>> import logging
>>> 
>>> f = foo()
>>> _f = dill.dumps(f)  
>>> f_ = dill.loads(_f)
>>> f_.logger
<logging.Logger object at 0x110b1d250>

View more solutions

Comments

[Pricey]

Perhaps you could link to (or quote) the exact answer you used? (There are two) There's a "link" button beneath any answer you can click for the url.

[Nathan McCoy]

this is an overlooked but nice response.

[Urchin]

Just a note, Loggers can be pickled; however, all it's doing is calling getLogger again. So if you're using multiprocesses, you still need to configure Logging prior to using the spawned Processes Logger.