How do I filter on tags in syslog-ng when they don't seem to be available by the time it processes it?

Question

I am having a hard time debugging an issue and I think it may be two-fold - problems in two separate programs.

The main issue is that I'm logging nginx to syslog on Ubuntu 14.04 like so:

access_log syslog:server=unix:/dev/log,tag=nginx,facility=local7,severity=info combined

I wanted to be able to filter messages that were tagged nginx, but no such luck.

filter nginx { facility(local7) and tags("nginx") }

Changing the and to an or works, or simply removing the tags portion, as the facility filter works fine. The tags filter does not, however.

So, I wanted to test using logger and setup a basic test as such:

template nginx { template("timestamp=${ISODATE} host=${HOST} tags=${TAGS} msgheader=${MSGHDR} ${MSG}\n"); template-escape(no); };

filter nginx { tags("nginx"); };

destination nginx { file("/tmp/nginx.log" template(nginx)); };

log { source(s_net); filter(nginx); destination(nginx); };

--

$ logger -n localhost -P 10001 -t nginx -p local7.info -u /tmp/ignored testing 123

I have a source, s_net, listening on UDP on port 10001 for this test. Using logger, I am logging to that port via UDP, tagging with "nginx" with local7 facility and severity of info, and, again, the filtering does not work. Removing the filter constraint let the message pass through to the destination, the same as it did with nginx directly. The tag just doesn't work.

Furthermore, the tag doesn't show up in the $TAGS macro, but instead in the $MSGHDR macro.

Note: /tmp/ignored is a workaround in logger < 2.0.2 whereby if you don't provide a socket it won't write to TCP/UDP ports at all, so providing a dummy flag lets it work

It seems that even when using logger, I cannot filter on incoming tag.

I am relatively new to syslog (and syslog-ng, specifically), so perhaps there is something I am missing? Documentation seems to be sparse, but this looks like a pretty simple use-case. Does anyone see anything particularly wrong or am I perhaps misunderstanding the purpose of tags?

Solution 1

There is a difference between the TAG field defined in the RFC3164 to which the nginx config parameter refers to and the tag filter used inside Syslog-ng.

The TAG field you can define in Nginx is interpreted as the program or the process which is logging the current message. It is in the MSG Part of the syslog packet and terminated by the first non-alphanumeric character. Everything after that will be used as the actual message (RFC3164#section-4.1.3).

You can see that when you sniff the syslog traffic e.g. with tcpdump:

tcpdump -A -vvv -s0 -n -i venet0 port 514

This is an example comming from NGINX with the config listed below:

15:25:16.477717 IP (tos 0x0, ttl 63, id 22206, offset 0, flags [DF], proto UDP (17), length 454) loadbalancer.example.com.45470 > log.example.com.514: [udp sum ok] SYSLOG, length: 426
    Facility local6 (22), Severity notice (5)
    Msg: Mar 10 15:25:16 loadbalancer.example.com nginx_access: "[10/Mar/2017:15:25:16 +0100]" "NO-CACHE" "app.example.com:80" "0.032" "302" "331" "10.235.121.191" "sub.example.com" "GET /location/?parameter=value1 HTTP/1.1" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Win64; x64; Trident/7.0; .NET CLR 2.0.50727; SLCC2; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET4.0E; Tablet PC 2.0)"

I sometimes use the TAG Field of nginx to distinguish between different logfiles:

access_log      /var/log/nginx/www.example.com.ssl_access_log proxy;
error_log       /var/log/nginx/www.example.com.ssl_error_log warn;

access_log       syslog:server=10.0.80.110,facility=local6,tag=nginx_access,severity=notice proxy;
error_log       syslog:server=10.0.80.110,facility=local6,tag=nginx_error,severity=error warn;

And on the receiving syslog-ng server I can then sort the incoming messages or just dump them in one big file:

source s_net { udp(); };

filter f_prg_nginx_access{ program(nginx_access); };
filter f_prg_nginx_error{ program(nginx_error); };

destination d_lb_access { file("/var/log/lb_access.log" perm(0640));};
destination d_lb_error { file("/var/log/lb_error.log" perm(0640)); };

log { source(s_net); filter(f_prg_nginx_access); destination(d_lb_access); };
log { source(s_net); filter(f_prg_nginx_error); destination(d_lb_error); };

As Robert Fekete already mentioned the tag filtering of Syslog-ng is another way for internaly tagging messages which arrive on a specific port or match a pattern. Additionally [2] says that tagging is the fastest way of sorting messages in syslog-ng but the tags are only locally available and not sent over the network.

[1] https://www.nginx.com/resources/admin-guide/logging-and-monitoring/ [2] https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guides/en/syslog-ng-ose-v3.6-guide-admin/html/tagging-messages.html

Solution 2

tags within syslog-ng are internal tags that you can explicitly add to the message (for example, at the source), or parse from the message.

I don't know how the nginx tag in logger or the access_log configuration affects the message itself, but I guess syslog-ng does not parse it.

Without seeing a sample message, I can suggest the following possible workarounds for the problem:

1. If the PROGRAM field of the messages is set, you can filter on that instead of the tag.
2. You mentioned that the tag appears in the MSGHDR macro. You can also filter on that, using the match() filter, something like: filter demo_filter { match("nginx" value("MSGHDR")) };
3. Send nginx messages to a separate port of your logserver, and on the logserver, configure a source to receive messages only on this port. syslog-ng will automatically add a tag for every message received from the source (see http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guides/en/syslog-ng-ose-v3.6-guide-admin/html/tagging-messages.html )

HTH

Regards,

Robert

Comments

[Jason McClellan]

Pretty old question, but this answer adds a lot of clarity. Thanks!