SyslogNG-How to optimise filter and log statements?

Question

Below is the current configuration for Syslog-NG logging, locally,

source s_network {
        udp(
                flags(syslog_protocol)
                keep_hostname(yes)
                keep_timestamp(yes)
                use_dns(no)
                use_fqdn(no)
        );
};

destination d_all_logs {
        file("/app/syslog-ng/custom/output/all_devices.log");

};

log {
        source(s_network);
        destination(d_all_logs);
};

---

To forward certain messages... below is the configuration to be added.

filter message_filter_string_1{ 
            match("01CONFIGURATION\/6\/hwCfgChgNotify\(t\)", value("MESSAGE"));
            }


filter message_filter_string_2{
            match("01SHELL\/5\/CMDRECORD", value("MESSAGE"));
            }

filter message_filter_string_3{
            match("10SHELL", value("MESSAGE"));
            }

filter message_filter_string_4{
            match("ACE-1-111008:", value("MESSAGE"));
            }

destination remote_log_server {
 udp("192.168.0.20" port(25214));
};

log { source(s_network); filter(message_filter_string_1); destination(remote_log_server); };

log { source(s_network); filter(message_filter_string_2); destination(remote_log_server); };

log { source(s_network); filter(message_filter_string_3); destination(remote_log_server); };

log { source(s_network); filter(message_filter_string_4); destination(remote_log_server); };

---

Actually there are more than 80 such filters

Does Syslog-NG config allow writing a syntax with single filter statement having match of regex1 or regex2 or regex3?

(or)

Does Syslog-NG config allow writing a syntax with single log statement having multiple filter?

Solution

If you want to combine multiple match statements, use or:

filter send_remote { 
            match("01CONFIGURATION\/6\/hwCfgChgNotify\(t\)", value("MESSAGE")) 
  or
            match("01SHELL\/5\/CMDRECORD", value("MESSAGE")) 
  or
            match("10SHELL", value("MESSAGE"))
  or
            match("ACE-1-111008:", value("MESSAGE"));

            }

... and then use that filter name once:

log { source(s_network); filter(send_remote); destination(remote_log_server); };

Comments

[Jeff Schaller]

In my opinion (not an Answer), you're not running afoul of DRY, since you're specifically filtering different messages. You've named the filters similarly, but that's it. Name it filter_message_ACE-1-111008, for example.

[overexchange]

@JeffSchaller Does Syslog-NG config allow writing a syntax with single filter statement having match of string1 or string2 or string3? (or) Does Syslog-NG config allow writing a syntax with single log statement having multiple filter?

[Jeff Schaller]

syslog-ng.com/technical-documents/doc/…

[overexchange]

What is the impact on performance of syslog-NG? Can we enhance performance in filtering every message with 80 matches in send_remote?

[Jeff Schaller]

I cannot test the performance impact in your environment; try it and see? You asked if syslog-ng could combine filter statements, and it appears that it can.

[overexchange]

here, it says: It is advised to use the simplest filters when filtering incoming messages. If a message can be filtered with several types of filters, check the measured data. For example, if a message is filtered with a regexp, the performance of syslog-ng can drop down to 85% of the original performance level. Whereas if the tag or facility filters are used, the decrease in performance is between 1-5%.

[Robert Fekete]

Hi, are these log messages that you are trying to match all from the same application? Can you post some sample log messages? Maybe there is an easier way to filter on them.