Extracting text from file using bash

Question

I am new to Linux and have a very large text log file from which to extract. I thought to use bash?

For example, the file contains:

Node:xyz
Time:01/07/13 14:26:17
INFO: Trusted certif ok

Node:abc
Time:01/07/13 14:26:18
INFO: Trusted certif ok

Node:def
Time:01/07/13 14:26:18
INFO: Trusted certif not ok

I need to extract the text after Node: and add it to the text after Info: to display on one line, output to be redirected to a new file. I am trying awk and sed, but not figured it out yet. Help much appreciated.

Example output would look like:

xyz Trusted certif ok
abc Trusted certif ok
dbf Trusted certif not ok

Solution 1

Try doing this :

in awk

awk -F: '/^Node/{v=$2}/^INFO/{print v $2}' file.txt

in bash :

while IFS=: read -r c1 c2; do
    [[ $c1 == Node ]] && var=$c1
    [[ $c1 == INFO ]] && echo "$var$c2"
done < file.txt

in perl :

perl -F: -lane '
    $v = $F[1] if $F[0] eq "Node";
    print $v, $F[1] if $F[0] eq "INFO"
' file.txt

in python (in a file, Usage : ./script.py file.txt ):

import sys
file = open(sys.argv[1])
while 1:
    line = file.readline()
    tpl = line.split(":")
    if tpl[0] == "Node":
        var = tpl[0]
    if tpl[0] == "INFO":
        print var, tpl[1]
    if not line:
        break

Solution 2

Using sed:

sed -n '/^Node/N;/Time/N;s/^Node:\([^\n]*\)\n[^\n]*\n[^ ]* /\1 /p' input

Comments

[Allen]

Thank you all very much. awk is awesome, as is your help.

[TrueY]

@EdMorton: what is the problem with the pure bash solution?

[Ed Morton]

@TrueY shell is an environment from which to call tools. It has programming language constructs (loops, etc.) to help you sequence the order in which you call tools. It is not a tool for parsing text files and so it's capabilities for doing that are extremely limited and it's side-effects non-obvious. For example, courtesy of the missing -r argument for read the script you posted will incorrectly interpret backslashes, and the use of echo will only work on some systems with some inputs. There may be additional edge cases it fails for and it's over twice the length of the robust awk script.

[TrueY]

@EdMorton: Thx 4 the answer! Generaly you are right. Every shell, interpreter and compiler has its own strength and weakness. The specific case will decide which tool to use. IMHO if some tool os not needed, it is better not to use. So in this case I prefer the pure bash version.

[Gilles Quenot]

Added -r switch for read in the bash solution to my edited POST.

[Ed Morton]

@TrueY IMHO you never need "some tool" as you can do any job entirely in shell, but you should always just use the right tool for each job. Avoiding using the tool that was created for the specific job you have to do makes no more sense in software than it would in any other job, e.g. when building a house and avoiding using the power drill that's sitting in your toolbox since you can put in screws using a manual screwdriver - yeah you can do it but it'll take you a lot longer.

[TrueY]

@EdMorton: Understood! Thx!

[Ed Morton]

@TrueY see why-is-using-a-shell-loop-to-process-text-considered-bad-pra‌​ctice for a great explanation of why the bash solution should be avoided.

[canfiese]

@GillesQuenot I know I'm a few years late to the party, but I tried the awk solution here and couldn't seem to get it to work. It's as if the variable "v" is recognized, but only the "Trusted certif ok" portion will print (if I ask it to print "v" on its own, it will print those values no problem). It's like awk ignores the first variable entirely. Is this because the behavior of awk has changed since you wrote this answer?

[Gilles Quenot]

@canfiese: try gawk