How to make apache output its logs in JSON instead of its default logging format?

Question

I need to display statistics from log files on my custom GUI. The log files are expected to be huge so parsing them manually would be overkill. However, if there is a way to output the logs in JSON instead of simple text, it could save a lot of time. Is there a way to do that?

Solution 1

Here is how I do, I learned it from somewhere on the internet,

For Errorlog:

ErrorLogFormat "{ \"time\":\"%{%Y-%m-%d}tT%{%T}t.%{msec_frac}tZ\", \"function\" : \"[%-m:%l]\" , \"process\" : \"[pid %P:tid %T]\" , \"message\" : \"%M\" ,\ \"referer\"\ : \"%{Referer}i\" },"

For Accesslog:

LogFormat "{ \"time\":\"%{%Y-%m-%d}tT%{%T}t.%{msec_frac}tZ\", \"process\":\"%D\", \"filename\":\"%f\", \"remoteIP\":\"%a\", \"host\":\"%V\", \"request\":\"%U\", \"query\":\"%q\", \"method\":\"%m\", \"status\":\"%>s\", \"userAgent\":\"%{User-agent}i\", \"referer\":\"%{Referer}i\" }," combined

Take care when adding some new variable, a minor mistake will stop your apache instance to invoke and make a backup of http.conf before trying.

Solution 2

The ErrorLogFormat did not work for me - I'm not sure if this is because I'm using Apache V2.4. I ended up with these formats:

ErrorLogFormat "{ \"time\":\"%{cu}tZ\", \"function\" : \"[%-m:%l]\" , \"process\" : \"[pid %P:tid %T]\" , \"message\" : \"%M\"}"

I found the format codes from the docs (https://httpd.apache.org/docs/2.4/mod/core.html#errorlogformat)

The LogFormat is as above:

LogFormat "{ \"time\":\"%{%Y-%m-%dT%T}t.%{usec_frac}t%{%z}t\", \"remoteIP\":\"%a\", \"host\":\"%V\", \"requestPath\":\"%U\", \"query\":\"%q\", \"method\":\"%m\", \"status\":\"%>s\", \"userAgent\":\"%{User-agent}i\", \"referer\":\"%{Referer}i\" }" json

Which is documented in the module (as opposed to the error log format, which is documented in the core, because it is not a module)

Comments

[IanB]

You can use \"time\":\"%{%Y-%m-%d}tT%{%T}t.%{msec_frac}tZ\" to get a JSON friendly timestamp e.g. 2017-07-07T08:41:54.380Z

[sshevlyagin]

The time format didn't work for me and I was getting empty referers, so here's what I used ErrorLogFormat "{ \"time\":\"%{%usec_frac}t\", \"function\" : \"[%-m:%l]\" , \"process\" : \"[pid %P]\" , \"outer\" : %M\ }"

[faintsignal]

How do you ensure that the message component of the error log is safely escaped for JSON encoding?

[Matthew Dutton]

Per mod_log_config docs: For security reasons, starting with version 2.0.46, non-printable and other special characters in %r, %i and %o are escaped using \xhh sequences, where hh stands for the hexadecimal representation of the raw byte. Exceptions from this rule are " and \, which are escaped by prepending a backslash, and all whitespace characters, which are written in their C-style notation (\n, \t, etc).

[Rax]

Backslash sequences in the path, or possibly other fields, cause broken JSON.