Parse a log4j log file

Question

We have several applications that use log4j for logging. I need to get a log4j parser working so we can combine multiple log files and run automated analysis on them. I'm not looking to reinvent the wheel, so can someone point me to a decent pre-existing parser? I do have the log4j conversion pattern if that helps.

If not, I'll have to roll our own.

Solution 1

I didn't realize that Log4J ships with an XML appender.

Solution was: specify an XML appender in the logging configuration file, include that output XML file as an entity into a well formed XML file, then parse the XML using your favorite technique.

The other methods had the following limitations:

• Apache Chainsaw - not automated enough
• jdbc - poor performance in a high performance distributed app

Solution 2

You can use OtrosLogViewer with batch processing. You have to:

1. Define you log format, you can use Log4j pattern layout parser or Log4j XmlLayout
2. Create java class that implements LogDataParsedListener. Method public void logDataParsed(LogData data, BatchProcessingContext context) will be called on every parsed log event.
3. Create jar
4. Run OtrosLogViewer with specifying your log processing jar, LogDataParsedListener implementation and log files.

Solution 3

Log4j log files aren't really suitable for parsing, they're too complex and unstructured. There are third party tools that can do it, I believe (e.g. Sawmill).

If you need to perform automated, custom analysis of the logs, you should consider logging to a database, and analysing that. JDBC ships with the JdbcAppender which appends all messages to a database of your choice, but it has performance implications, and it's a bit flaky. There are other, similar, alternatives on the interweb, though (like this one).