Multicolored Grep

Question

I'm trying to get each grep command to highlight its results in a different color. I can do it manually with a line like this:

ls -l GREP_COLORS='mt=01;32' grep c | GREP_COLORS='mt=01;31' grep o | GREP_COLORS='mt=01;34' grep n | GREP_COLORS='mt=01;36' grep f

Every c character will be highlighted in green and every o character will be highlighted in red, etc...

For this example to work you'll need to ensure that you always have --color=always on your grep commands. I've set this in my .bashrc so grep will always have colors:

export GREP_OPTIONS='--color=always'

---

What I'm attempting to accomplish is to wrap this functionality with an alias, so I can just call grep and have a different GREP_COLORS value each time. I understand the consideration of multiple shells for each new piped grep and I'm trying to over come this by creating some files (one for each color), to indicate that they have already been used.

I have made some attempts but strangely, this one seems to work the "best". I have this in my .bashrc:

alias mg="mygrep"
mygrep(){
    # define possible colors
    COLORS=("01;32" "01;31" "01;34" "01;36")
    COUNTER=0
    NUM=0
    # as long as the color has already been used, keep searching
    while [ -f /home/lior/Desktop/mygrep_$NUM ]; do
        # get a random index
        let NUM=`shuf --input-range=0-$(( ${#COLORS[*]} - 1 )) | head -1`
        wait ${!}
        $(( COUNTER+=1 ))
        if [ "$COUNTER" -ge ${#COLORS[@]} ]; then
            # remove all color locks
            rm /home/lior/Desktop/mygrep_*
            wait ${!}
        fi
    done
    # mark this color as used
    touch /home/lior/Desktop/mygrep_$NUM
    wait ${!}

    # lets go!
    GREP_COLORS="mt=${COLORS[$NUM]}" grep "$@"
}

I'm using this alias like so:

ll | mg c | mg o | mg n | mg f

The results are quite cool. There are however some errors that are slightly different each time. Here are a couple of screenshots:

Looks like as the shell goes through each pipe command, the previous function did not yet finish its execution. It tries to remove files that don't exist anymore. I'm not to sure where those other command not found errors are coming from.

As you can see, I've put in some wait commands to try let the file manipulation complete but this doesn't seem to be working too well. Another thing I have already tried is to use shared memory /dev/shm but it yielded similar results.

How would I go about getting the results I want?

Note:

I am looking for answers that simply wrap the grep command as it has lots of functionality that I'm wanting to use and intend to insert other logic between the pipes, so I don't want to provide all of the search terms at once. I'm also not looking for other "grep like" tools. Sorry to @terdon who has already posted an awesome perl suggestion.

Solution 1

Here's a different approach. I have a little Perl script which I have already posted in another answer that will highlight the user provided patterns in different colors. A slightly modified version of the script will act like grep:

#!/usr/bin/env perl
use Getopt::Std;
use strict;
use Term::ANSIColor; 

my %opts;
getopts('hic:l:',\%opts);
    if ($opts{h}){
      print<<EoF; 
Use -l to specify the pattern(s) to highlight. To specify more than one 
pattern use commas. 

-l : A Perl regular expression to be colored. Multiple expressions can be
     passed as comma separated values: -l foo,bar,baz
-i : makes the search case sensitive
-c : comma separated list of colors;

EoF
      exit(0);
    }

my $case_sensitive=$opts{i}||undef;
my @color=('bold red','bold blue', 'bold yellow', 'bold green', 
       'bold magenta', 'bold cyan', 'yellow on_blue', 
       'bright_white on_yellow', 'bright_yellow on_red', 'white on_black');
if ($opts{c}) {
   @color=split(/,/,$opts{c});
}
my @patterns;
if($opts{l}){
     @patterns=split(/,/,$opts{l});
}
else{
    $patterns[0]='\*';
}

# Setting $| to non-zero forces a flush right away and after 
# every write or print on the currently selected output channel. 
$|=1;

while (my $line=<>) 
{ 
    my $want=0;
    for (my $c=0; $c<=$#patterns; $c++){
    if($case_sensitive){
        if($line=~/$patterns[$c]/){
           $line=~s/($patterns[$c])/color("$color[$c]").$1.color("reset")/ge;
           $want++;
        }
    }
    else{
        if($line=~/$patterns[$c]/i){
          $line=~s/($patterns[$c])/color("$color[$c]").$1.color("reset")/ige;
          $want++;
        }
      }
    }
print STDOUT $line if $want>0;
}

If you save that script as cgrep somewhere in your PATH and make it executable, you can specify up to 10 different patterns, each of which will be printed in a different color:

$ cgrep -h
Use -l to specify the pattern(s) to highlight. To specify more than one 
pattern use commas. 

-l : A Perl regular expression to be colored. Multiple expressions can be
     passed as comma separated values: -l foo,bar,baz
-i : makes the search case sensitive
-c : comma separated list of colors;

Solution 2

Each invocation of grep in a pipe runs in a separate shell, so you'll need to pass some state between them. The following solution is a crude way to handle that with a file that keeps the colour index and a lock file which ensures that simultaneous calls do not read the same value:

#!/usr/bin/env bash
color_index_file=~/.gitcolor
color_index_lock_file=/tmp/$(basename $0)

colors=()
for index in {31..34}
do
    colors+=("01;$index")
done

until mkdir "$color_index_lock_file" 2>/dev/null
do
    :
done

color_index=$(($(cat "$color_index_file" || echo 0) + 1))

if [[ $color_index -ge ${#colors[@]} ]]
then
    color_index=0
fi

printf "$color_index" > "$color_index_file"
rmdir "$color_index_lock_file"

GREP_COLORS="mt=01;${colors[$color_index]}" grep --color=always "$@"

Test assuming you've named your copy cgrep and put it on your PATH:

echo foobarbaz | cgrep foo | cgrep bar | cgrep baz

Solution 3

If you're good with regular expressions, You may want to check out grc and grcat. grc calls grcat.

grcat uses configuration files where you can add regular expressions to match text to be displayed in each color. It also has a number of other options. It defaults to colorizing system log files.

Depending on what you have in mind for your final script, you may be able to colorize your output with just one command.

The trick is specifying the regexes correctly for each "field" in your data source. This will be a lot easier if your data is relatively uniform in structure.

Last time I tried it, I didn't get too far, but I may have another go at it because I'm a bit better at regexes than I was then.

There is also the tput command which can be used to send information (like color changes) directly to your terminal device.

Both approaches are covered by the post below. It talks about the find command, but it can be applied to the output of any command.

Colored FIND output?

Comments

[Bernhard]

Your first example does not match what you are explaining in the remaining part, just saying.

[Lix]

@bernhard I don't understand what you mean.. My intention is to use my alias as part of a bigger command using piping... Please let me know what contradiction you are speaking of...

[Bernhard]

I thought you wanted to use your alias outside of pipes too. Anyhow, your first example does not work for me. Did you try alias mg="mygrep; grep"?

[Lix]

@Bernhard - I'm working on an ubuntu 12.04 box. I wouldn't be surprised if there were slight differences... I did try your suggestion, the problem with that is that mygrep; turns into a new command in itself and the data stream get's lost. The incoming pipe from the ls would get passed to mygrep; and not to grep. At least that is how I understand it.

[Lix]

@Bernhard - Ah.. I think I know why it didn't work for you. You need to make sure that you have --color=always on all your grep commands. I've set that globally in my .bashrc. I've edited that into the post.

[Bernhard]

got it to work everywhere, except after pipes, good question :)

[Emmanuel]

Surprisingly you will notice that most of the time the information you seek in the line is not what has matched the pattern but the information that has not matched the pattern.

[Lix]

@Emmanuel - Yes, but I would like to know why that specific line is being processed IE - what grep command matched against it.

[Emmanuel]

@Lix Ok, that can be used to detect false positives.

[goldilocks]

Cool idea. alias has some limitations as you notice. I'd just encapsulate this in a script, call it mygrep or whatever and put it in ~/bin, presuming that's in your $PATH.

[Lix]

The only variables that really need to be maintained are COLOR_INDEX and GREP_COLORS. I tried exporting these at the end of the function without success. Is that what you meant? Simply to have export VAR, right?

[Lix]

Oh - and yea.. silly typo there with COLOR_TOGGLE. Thanks for catching it :)

[Bernhard]

@l0b0 The exporting does not work for me either, have to downvote for now until it really answers the question.

[l0b0]

@Bernhard Does this work for you?

[Lix]

Thanks for your input! I've incorporated your grep "$@" suggestion - that sorted out the alias to run the function and then grep.

[Lix]

However both of our solutions suffer from the same problem; When doing multiple pipes, only one color is used. Only by manually specifying each color (as I have done in my first snippet) is the output using more than one color.

[l0b0]

@Lix What do you mean only one color is used? Do you have an example?

[Lix]

@l0b0 I've updated my post. It's slightly modified from the original version. Thanks for your help so far!

[Bernhard]

@l0b0 Yup, this seems to do what is intended.

[BhishanPoudel]

And, how to get .gitcolor file ?