Capture piped multiline grep output into a variable

Question

I'm using Zabbix's zabbix_sender.sh script to push exception stack traces to my remote Zabbix monitoring server.

zabbix_sender.sh requires a key and a value for any data that it sends. It can read data from stdin, but this overrides any key variable specified. Because my stdin data isn't formatted like Zabbix expects, I need to pass in "value" as an argument. Hopefully this provides some context.

What I want to accomplish is capturing the multiline result from grep into a variable, preserving newlines, so that I can call the zabbix_sender.sh script with that variable as an argument.

What I've tried so far looks like this:

tail -Fn0 /var/log/uwsgi.log | grep "Exception:" -A 100 | (read tback; /usr/local/zabbix/bin/zabbix_sender -z myzserver.com -s MyHostName -k uwsgi_traceback -o $tback)

As best I can tell, that never invokes zabbix_sender.sh.

To test, I've tried using this command, which doesn't seem to work either:

tail -Fn0 /var/log/uwsgi.log | grep "Exception:" -A 100 | (read errorlines; echo "$errorlines" > /tmp/errorlines.txt)

The /tmp/errorlines.txt file is never created.

How can I capture grep's output lines into a variable so that I can call another script with that variable as an argument?

Solution 1

The problem is the tail part; since it is in continuos mode it wont ever spit out something for "read" to read.

This should work:

#!/bin/bash
echo "0" >/tmp/numberoflines
IFS=''
while [ 1 ]
do
    NUMBER=$(cat /tmp/numberoflines)
    LINES=$(wc -l < /var/log/uwsgi.log)
    DIFFERENCE=$(($LINES-$NUMBER))

    if [ $DIFFERENCE != 0 ]; then
            exception=$(tail -n $DIFFERENCE /var/log/uwsgi.log | grep "Exception:" -A 100)
           /zabbix/bin/zabbix_sender -z myzserver.com -s MyHostName -k uwsgi_traceback -o $tback) $exception;
    fi

sleep 5;
echo "$LINES" >/tmp/numberoflines
done

Solution 2

Here my approach.

1. Use zabbix "log" type key to monitor error pattern in a log-file (/var/log/uwsgi.log is the case here).
2. invoke zabbix remote command triggered by the above 1. This remote command fetch lines surrounding the error by linux command tail(1).

The advantages of this approach are:

1. No need to install&setup agent-host side special script (like zabbix_sender.sh discussed above). zabbix 'log' type item has been already provided such a kind of purpose.
2. no zabbix_sender.sh-like script on agent-host mean that there is no additional CPU & memory consumption. Fetching several lines surounding error in the log happens only at trigger.

Let me explain the detail how to setup below:

1. register regular expression in zabbix from Administration > General > Regular expressions > [New regular expression] like "error|fail|fatal". Let's assume the variable name is @uwsgi_error_pattern.
2. register item from Configuration > Hosts > [Target host] line > Items > [Create Item] with the following attributes:
   • Description: [any name]
   • Type: Zabbix agent(active)
   • Key: log[/var/log/uwsgi.log,@uwsgi_error_pattern]
   • Type of information: Log
3. register 2nd item from Configuration > Hosts > [Target host] line > Items > [Create Item] with the following attributes, to accept fetched log flagment sent by zabbix remote command(mentioned later):
   • Description: [any name]
   • Type: Zabbix trapper
   • Key: my_app.fetch_uwsgi_log
   • Type of information: Text
4. register trigger from Configuration > Hosts > [Target host] line > Triggers > [Create trigger] with the following attributes:
   • Name: uwsgi log monitor on {HOSTNAME}
   • Expression ({[Target host]:log[/var/log/uwsgi.log,@uwsgi_error_pattern].iregexp(@uwsgi_error_pattern)})#0&({[Target host]:log[/var/log/uwsgi.log,@uwsgi_error_pattern].nodata(300)})=0
5. register action which remotely executes to fetch data surrounding the error line of the log file as follows:
   • Name: Fetch latest uwsgi log on error
   • Action Operations:
     • Operation Type: Remote command
     • Remote command: {HOSTNAME}:zabbix_sender -z [zabbix-server] -s {HOSTNAME} -k my_app.fetch_uwsgi_log -o "`tail -200 /var/log/uwsgi.log`"

NOTE-1: key name "my_app.fetch_uwsgi_log" in the above step is just example. We can define any unique name to bind zabbix_sender and the item.

NOTE-2: You may need AllowRoot=1 on /etc/zabbix/zabbix_agentd.conf to allow zabbix-agent read uwsgi.log.

Solution 3

You must turn off word splitting in bash, eg, by clearing IFS:

export IFS=""
set NEWVAR=`your tail|grep expression`

Now echo $NEWVAR has the newlines.

Comments

[Michael Hampton]

Why don't you use log file monitoring?

[skyler]

Zabbix's built in log file monitoring is great at alerting me that something went wrong, but I haven't managed to get it to record any surrounding lines. Even using the output argument to log, I can't seem t make it capture more than the line that triggered it. If you know of a way to make this work, please share it with me.

[skyler]

This will run once and capture tail's output, no? Will it follow the file as it's written to by the app that I'm logging?

[skyler]

Unfortunately that didn't work export IFS="" tail -Fn0 /var/log/uwsgi.log | grep "Exception:" -A 100 | (read errorlines; echo "$errorlines" > /tmp/errorlines.txt)

[pdwalker]

I marked up this answer because I didn't know about the IFS trick for capturing multiline output into a variable.

[Fumisky Wells]

Is $exception just before "fi"(end of "if") typo?