log4j2 - Syslog appender and PatternLayout

Question

I need to log events into the syslog. I use lo4j2 and the syslog appender. My appenders block in log4j2.xml looks like this:

<appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
        </Console>
        <Syslog name="syslog" host="localhost" port="514" protocol="UDP" charset="ISO-8859-1">
        </Syslog>
        <RollingFile name="AppLog" fileName="/var/log/app.log"
                     filePattern="/var/log/$${date:yyyy-MM}/app-%d{MM-dd-yyyy}-%i.log.gz">
            <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
            <Policies>
                <TimeBasedTriggeringPolicy/>
            </Policies>
        </RollingFile>          
    </appenders>

As you can see I have a Console appender and RollingFile appender with a specific PatternLayout. I want to use the same PatternLayout for the Syslog appender. However, the log messages in the syslog seem to always use a predefined layout. I tried to do the following:

<Syslog name="syslog" host="localhost" port="514" protocol="UDP" charset="ISO-8859-1">
    <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
</Syslog>

But this does not have any effect. the syslog messages still have the same predfined format.

How can I determine the format of my log messages that go into the syslog?

Solution 1

As mentioned in this log4j2 bug report, the developers of log4j2 coded the SyslogAppender as a SocketAppender hardwired to a SyslogLayout

because it is intended to conform to either the original syslog format or RFC 5424. No other Layout should be permitted.

They unfortunately did not realize that the RFC 5424 specifications do not enforce any particular format for the message contained in the log, that in the Log4j2 implementation is only the %m portion of the log.

To solve this issue, a solution (suggested in the same bug report) is to reproduce the syslog format using a PatternLayout inside a SocketAppender, like so

<Socket name="SYSLOG" host="localhost" port="514" protocol="UDP">
  <PatternLayout
    pattern="&lt;1&gt;%d{MMM dd HH:mm:ss} ${hostName} appName: {
      &quot;host&quot;:&quot;${hostName}&quot;,
      &quot;thread&quot;:&quot;%t&quot;,
      &quot;level&quot;:&quot;%p&quot;,
      &quot;logger&quot;:&quot;%c{1}&quot;,
      &quot;line&quot;:%L,
      &quot;message&quot;:&quot;%enc{%m}&quot;,
      &quot;exception&quot;:&quot;%exception&quot;
      }%n"
  />
</Socket>

This will write well-formatted RFC5424 logs to local 514 port through UDP. Following is a sample log output:

Sep 14 10:40:50 app-hostname app-name: { "host":"host-name-01", "thread":"http-nio-8080-exec-4", "level":"DEBUG", "logger":"ExecuteTimeInterceptor", "line":52, "message":"GET /health 200 served in 3", "exception":"" }

Solution 2

I don't believe you can use a pattern on the basic Syslog appender.

From the docs it states that

"SyslogAppender is a SocketAppender that writes its output to a remote destination specified by a host and port in a format that conforms with either the BSD Syslog format or the RFC 5424" http://logging.apache.org/log4j/2.x/manual/appenders.html#SyslogAppender

However, it does allow you to specify "format = RFC 5424"

If you use RFC 5424

Then you can put a PatterLayout in the loggerFields parameter. See http://logging.apache.org/log4j/2.x/manual/layouts.html#RFC5424Layout

Hope that helps!

Solution 3

You can use add additional elements to an RFC5424 formatted SyslogAppender message using the LoggerFields tag like this:

<LoggerFields>
  <KeyValuePair key="thread" value="%t"/>
  <KeyValuePair key="priority" value="%p"/>
  <KeyValuePair key="category" value="%c"/>
  <KeyValuePair key="exception" value="%ex"/>
</LoggerFields>

I then pull these out using rsyslog's RFC5424 parsing module, mmpstrucdata, to create json tree. The rsyslog.conf template for accessing them looks like:

template(name="jsondump" type="string" string="'%$!rfc5424-sd!mdc@18060!thread%', '%$!rfc5424-sd!mdc@18060!priority%', '%$!rfc5424-sd!mdc@18060!category%', '%$!rfc5424-sd!mdc@18060!exception%'")

I was just trying to do the same and thought I'd share what worked for me. - Sam

Solution 4

You can use SocketAppender and PatternLayout to format syslog (syslog-ng) messages.
To support dynamic Severities with a fixed Facility (e.g.: 'user-level messages' - see RFC5424) the pattern should look like this:

<Socket name="SYSLOG" host="${env:INTERFACE}" port="514" protocol="UDP">
   <PatternLayout pattern="&lt;%level{TRACE=15, DEBUG=15, INFO=14, WARN=12, ERROR=11, Fatal=11,&gt;%replace{${env:APPLICATION_NAME}}{\r}{}[%X{PID}] %t(%T) %c{10} - %m%n"/>
</Socket>

To calculate the Priority value (PRIVAL) for Facility 'user-level message' and Severity 'informational messages' - see [RFC5424](https://www.rfc-editor.org/rfc/rfc5424/)) the following example might help:

Syslog:          Facility  | Severity
Numerical Code:      1          6
Bin:             0 0 0 0 1 |  1 1 0
Dec:                 8     +    6    =  14

log4j2syslog-ngsocketappenderpatternlayout

Solution 5

I used the config posted by butcher82, but had to change it a bit to produce the result I needed.

What I got in the end is a message with the correct priority, timestamp (without leading zeros for days), host and a message part. The mapping between syslog and log4J level is used as defined in org.apache.log4j.Level and the facility is set to 1 (user-level messages), to simplify the priority calculation.

This pattern should be compatible with RFC-3164:

<Socket name="SysLogAppender" host="localhost" port="514" protocol="UDP">
    <PatternLayout pattern="&lt;%level{TRACE=7, DEBUG=7, INFO=6, WARN=4, ERROR=3, Fatal=0}&gt;%d{MMM d hh:mm:ss} ${hostName} %m%n"/>
</Socket>

Below is the produced output:

<3>Dec 15 09:59:16 foo.bar.hostname this is a test message

Note: One might add an application name or pid, after the hostname.

View more solutions

Comments

[Doron Gold]

This seems like the right direction, but I don't quite understand how to use the <loggerFields> tag in my log4j2.xml. what should I put in the 'key' attribute? An example would be very useful.

[philnate]

@conornicol, I believe you misunderstand the PatternLayout part. It's by far not possible to inject a custom PatternLayout into the RFC-5424 Layout. What you're influencing is the part within []. You're not affecting the message part of that layout.

[Ankush Tehale]

Is there way to add these LoggerFields in log4j2.properties files. I want to do it for elasticsearch 5x and it support only log4j.properties no log4j2.xml and I don't see example any where on net and docs for LoggerFields in properties file

[Ankush Tehale]

Adding this to properties files does not work appender.sumo_syslog.loggerFields={'key' : 'priority', 'value' : '%p'}

[modle13]

This does not allow writing messages to a custom facility, such as local4.

[Michail Alexakis]

You will also need to load the proper rsyslog module to be able to parse RFC5424 struct: module(load="mmpstrucdata") action(type="mmpstrucdata")