what is the proper way to do logging in csv file?

Question

i want to log some information of every single request send to a busy http server in a formatted form,use log module would create some thing i don't want to:

[I 131104 15:31:29 Sys:34]

i think of csv format but i don't know how to customize it,and python got csv module,but read the manual

import csv
with open('some.csv', 'w', newline='') as f:
    writer = csv.writer(f)
    writer.writerows(someiterable)

since it would open and close a file each time, i am afraid in this way would slow down the whole server performance, what could i do?

Solution 1

Just use python's logging module.

You can adjust the output the way you want; take a look at Changing the format of displayed messages:

To change the format which is used to display messages, you need to specify the format you want to use:

import logging
logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG)
logging.debug('This message should appear on the console')
logging.info('So should this')
logging.warning('And this, too')

and Formatters:

Formatter objects configure the final order, structure, and contents of the log message.

You'll find a list of the attribtus you can use here: LogRecord attributes.

---

If you want to produce a valid csv-file, use python's csv module, too.

Here's a simple example:

import logging
import csv
import io

class CsvFormatter(logging.Formatter):
    def __init__(self):
        super().__init__()
        self.output = io.StringIO()
        self.writer = csv.writer(self.output, quoting=csv.QUOTE_ALL)

    def format(self, record):
        self.writer.writerow([record.levelname, record.msg])
        data = self.output.getvalue()
        self.output.truncate(0)
        self.output.seek(0)
        return data.strip()

logging.basicConfig(level=logging.DEBUG)

logger = logging.getLogger(__name__)
logging.root.handlers[0].setFormatter(CsvFormatter())

logger.debug('This message should appear on the console')
logger.info('So should "this", and it\'s using quoting...')
logger.warning('And this, too')

Output:

"DEBUG","This message should appear on the console"
"INFO","So should ""this"", and it's using quoting..."
"WARNING","And this, too"

Solution 2

I would agree that you should use the logging module, but you can't really do it properly with just a format string like some of the other answers show, as they do not address the situation where you log a message that contains a comma.

If you need a solution that will properly escape any special characters in the message (or other fields, I suppose), you would have to write a custom formatter and set it.

logger = logging.getLogger()

formatter = MyCsvFormatter()

handler = logging.FileHandler(filename, "w")
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.setLevel(level)

You'll obviously have to implement the MyCsvFormatter class, which should inherit from logging.Formatter and override the format() method

class MyCsvFormatter(logging.Formatter):
    def __init__(self):
        fmt = "%(levelname)s,%(message)s" # Set a format that uses commas, like the other answers
        super(MyCsvFormatter, self).__init__(fmt=fmt)

    def format(self, record):
        msg = record.getMessage()
        # convert msg to a csv compatible string using your method of choice
        record.msg = msg
        return super(MyCsvFormatter, self).format(self, record)

Note: I've done something like this before, but haven't tested this particular code sample

As far as doing the actual escaping of the message, here's one possible approach: Python - write data into csv format as string (not file)

Solution 3

As sloth suggests, you can easily edit the delimiter of the log to a comma, thus producing a CSV file.

Working example:

import logging

# create logger
lgr = logging.getLogger('logger name')
lgr.setLevel(logging.DEBUG) # log all escalated at and above DEBUG
# add a file handler
fh = logging.FileHandler('path_of_your_log.csv')
fh.setLevel(logging.DEBUG) # ensure all messages are logged to file

# create a formatter and set the formatter for the handler.
frmt = logging.Formatter('%(asctime)s,%(name)s,%(levelname)s,%(message)s')
fh.setFormatter(frmt)

# add the Handler to the logger
lgr.addHandler(fh)

# You can now start issuing logging statements in your code
lgr.debug('a debug message')
lgr.info('an info message')
lgr.warn('A Checkout this warning.')
lgr.error('An error writen here.')
lgr.critical('Something very critical happened.')