Python logging multiple files using the same logger

20,831

Solution 1

Finally I decided to define two loggers, because:

  • They are for different purposses. In my case, one logs input request to a web service, and the other one logs the response. And they use different files to it

  • I'm using a logging config file, in a frontal web service. Adding/removing handlers before logging messages is not the right approach, as @mike said. Thx to @drekyn too!

Here is my logging config file, just for reference if anyone is interested in:

[loggers]
keys=root, ws_in_log, ws_out_log

[handlers]
keys=consoleHandler, ws_in_hand, ws_out_hand

[formatters]
keys=generic_form

[logger_root]
handlers=consoleHandler
level=NOTSET

[logger_ws_in_log]
level=NOTSET
handlers=ws_in_hand
qualname=ws_in_log

[logger_ws_out_log]
level=NOTSET
handlers=ws_out_hand
qualname=ws_out_log

[handler_ws_in_hand]
class=logging.handlers.TimedRotatingFileHandler
level=NOTSET
formatter=generic_form
args=('/path/ws_in_.log', 'h', 1, 0, None, False, True)

[handler_ws_out_hand]
class=logging.handlers.TimedRotatingFileHandler
level=NOTSET
formatter=generic_form
args=('/path/em/ws_out_.log', 'h', 1, 0, None, False, True)

[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=generic_form
args=(sys.stdout,)

[formatter_generic_form]
format='%(asctime)s - %(levelname)s - %(message)s'
datefmt='%Y-%m-%d %H:%M:%S'
class=

See you!

Solution 2

You should instantiate an Handler for each destination you want to send your log to, then add the 2 handlers to your logger. The following should work (didn't test it though):

logger = logging.getLogger()
handler1 = logging.TimedRotatingFileHandler()
handler2 = logging.TimedRotatingFileHandler()
logger.addHandler(handler1)
logger.addHandler(handler2)

Of course add all your configuration and formatting options you may need. Basically it is just to show you that when you instantiate the logging handler you can add it to the logger. From that moment on, your log records will be emitted to every handler added to the logger.

Solution 3

what you want is to

  1. create 2 NON ROOT loggers.
  2. make handler for each one, point to different file
  3. add handler to appropriate logger

    logger1 = logging.getLogger('general_logger')
    logger2 = logging.getLogger('some_other_logger')
    
    log_handler1 = logging.handlers.RotatingFileHandler(file_1, *args)
    log_handler2 = logging.handlers.RotatingFileHandler(file_2, *args)
    
    logger1.addHandler(log_handler1)
    logger2.addHandler(log_handler2)
    

then

    logger1.info("this will be logged to file_1 ")
    logger2.info("this will be logged to file_2 ")

Please note that if you create a ROOT logger and a different logger, root logger will log everything that this different controller is trying to log.

In other words, if

    root_logger = logging.getLogger()
    logger2 = logging.getLogger('some_other_logger')

    root_log_handler = logging.handlers.RotatingFileHandler(file_1, *args)
    log_handler2 = logging.handlers.RotatingFileHandler(file_2, *args)

    root_logger.addHandler(root_log_handler)
    logger2.addHandler(log_handler2)

then

    root_logger.info("this will be logged to file_1 ")
    logger2.info("this will be logged to file_1 AND file_2 ")
Share:
20,831
Alberto Megía
Author by

Alberto Megía

:)

Updated on July 26, 2020

Comments

  • Alberto Megía
    Alberto Megía over 3 years

    This is my scenario: I want to log my_module's activity. This needs to be done, depending on the method executed (let's say, INPUT and OUTPUT), to two different files.

    So I have two Handlers, each one point to a different file (my_in_.log & my_out_.log), with the same log level. I would like to know if I can use the same logger to achieve this or I have to define two loggers. My config is:

    [loggers]
    keys=root, my_log
    
    [handlers]
    keys=my_in_hand, my_out_hand
    
    [formatters]
    keys=generic_form
    
    
    ...
    
    
    [logger_my_log]
    level=NOTSET
    handlers=my_in_hand, my_out_hand
    qualname=ws_log
    
    [handler_my_in_hand]
    class=handlers.TimeRotatingFileHandler
    level=NOTSET
    formatter=generic_form
    args=('my_in_.log', 'h', 1, 0, None, False, True)
    
    [handler_my_out_hand]
    class=handlers.TimeRotatingFileHandler
    level=NOTSET
    formatter=generic_form
    args=('my_out_.log', 'h', 1, 0, None, False, True)
    

    Do I have to define a logger per handler/destination (because I want to log different information in different files)? Is there a way to indicate to the logger which handler will do this? I mean, I have two handlers for one logger, then choose only one handler to log one method.

  • Alberto Megía
    Alberto Megía about 11 years
    Maybe I am not clear: I do not want to log my records to every handler added to the logger. What I want is to select which handler will log the record, that is, one logger -> two handlers but choose which destination the record will be allocated in. Reuse the same logger with different handlers, but some records will be logged by one handler, and some with another... Is this possible or I have to define other logger? THX A LOT! :)
  • mike
    mike about 11 years
    @AlbertoMegía You should define your own method that takes as argument the log destination and uses internally the right handler for the log message.
  • Alberto Megía
    Alberto Megía about 11 years
    So @mike you mean I have to add and remove logger's handler to select it in runtime? This may not be my best option because this module is my frontal view in a web service... with every request I would have to switch handlers... :S
  • mike
    mike about 11 years
    @AlbertoMegía well since this is a frequent action, adding/removing handlers before logging the message is not the right approach (obviously).
  • drekyn
    drekyn about 11 years
    which are the criteria for which you use one logger or the other? You might overwrite the emit() function of the handler in order to do this
  • Alberto Megía
    Alberto Megía about 11 years
    @drekyn Criteria: log input request to a file. Log output to another. Output records format != input records format
  • drekyn
    drekyn about 11 years
    I guess that in this case then you need to define 2 loggers with 2 handlers, one each. in this way you'll keep contexts separated and clear
  • Alberto Megía
    Alberto Megía about 11 years
    I think that will be the best approach, @drekyn. But I will wait, maybe somebody knows how to deal with it... THX A LOT GUYS! :)
  • Ogre Psalm33
    Ogre Psalm33 over 3 years
    I feel like other answers didn't understand exactly what you were trying to do. But, this answer (and your question) is also exactly what I'm trying to do!