Confused with syslog message format

Question

I am a bit confused about syslog message format. I have to write a program that parses syslog messages. When I read what I get in my syslog-ng instance I get messages like this:

Jan 12 06:30:00 1.2.3.4 apache_server: 1.2.3.4 - - [12/Jan/2011:06:29:59 +0100] "GET /foo/bar.html HTTP/1.1" 301 96 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 ( .NET CLR 3.5.30729)" PID 18904 Time Taken 0

I can clearly determine the real message (which is, in this case an Apache access log message) The rest is metadata about the syslog message itself.

However when I read the RFC 5424 the message examples look like:

without structured data

 <34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8

or with structured data

<165>1 2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] BOMAn application event log entry...

So now I am a bit confused. What is the correct syslog message format ? It is a matter of spec version where RFC 5424 obsoleted RFC 3164 ?

Solution 1

The problem in this case is that apache is logging via the standard syslog(3) or via logger. This only supports the old (RFC3164) syslog format, i.e. there is no structured data here. In order to have the fields from the apache log show up as RFC5424 structured data, apache would need to format the log that way.

The first example is not proper RFC3164 syslog, because the priority value is stripped from the header. Proper RFC3164 format would look like this:

<34>Jan 12 06:30:00 1.2.3.4 apache_server: 1.2.3.4 - - [12/Jan/2011:06:29:59 +0100] "GET /foo/bar.html HTTP/1.1" 301 96 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 ( .NET CLR 3.5.30729)" PID 18904 Time Taken 0

Traditionally rfc3164 syslog messages are saved to files with the priority value removed.

The other two are in RFC5424 format.

Solution 2

If you have access to the installed syslog-daemon on the system you could configure it to write the logs (received both locally or via network) in a different format. rsyslogd for instance allows to configure your own format (just write a template) and also if I remember correctly has a built-in template to store in json format. And there are libraries in almost any language to parse json.

EDIT: You could also make rsyslogd part of your program. rsyslog is very good in reading incoming syslogs in either of the two RFC formats. You can then use rsyslog to output the message in JSON. This way rsyslog does all the decompositioning of the message for you.

Comments

[qwix]

Ok so I believe this is normal to see almost any log messages in RFC3164 format instead of RFC5424 because it has been there for a very long time. Which also means I have to write a parser that support both specification. Do you know any library (no matter the language) that already do this ?

[b0ti]

Why not use an existing tool such as syslog-ng, rsyslog, or nxlog? (note that I'm affiliated with the latter). If you need to parse this from your own program, I'd probably pick a scripting language which can easily handle text files, such as perl or python.

[qwix]

I need to parse these messages from my own program. I think I am going to use Python. But for some reason parsing syslog messages does not seem as simple as I thought. Probably because I need to support two difference specifications. I believe it is the perfect time to start writing unit tests. Thanks for your answer :)