How to do a continous 'wc -l' with gnu texttools?

Question

I know of course that

cat logfile.txt | wc -l
120

will tell me the number of lines in a file.

Whereas

tail -f logfile.txt

will show me the new lines that another program writes to logfile.txt.

Is it possible to combine both so that I get a continuous updating line count of logfile.txt with standard text utilities?

I do know about

watch wc -l logfile.txt

but I do not want to re-count the whole file each time, that seems to be a waste. One would need an appended-only count every second or so and probably an \r instead of an \n at the end of line.

Solution 1

Maybe:

tail -n +1 -f file | awk '{printf "\r%lu", NR}'

Beware that it would output a number for every line of input (though overriding the previous value if sent to a terminal).

Or you can implement the tail -f by hand in shell:

n=0
while :; do 
  n=$(($n + $(wc -l)))
  printf '\r%s' "$n"
  sleep 1
done < file

(note that it runs up to one wc and one sleep command per second which not all shells have built in. With ksh93 while sleep is builtin, to get a built in wc (at least on Debian), you need to add /opt/ast/bin at the front of $PATH (regardless of whether that directory exists or not) or use command /opt/ast/bin/wc (don't ask...)).

You could use pv, as in:

tail -n +1 -f file | pv -bl > /dev/null

But beware that it adds k, M... suffixes when the number is over 1000 (and there doesn't seem to be a way around that).

Solution 2

Try to count it with pure bash without wc:

a=0 ; tail -f file | while read -r line ; do ((a++)) ; echo $a ; done

or even like this to rewrite previous value:

a=0 ; tail -f file | while read -r line ; do ((a++)) ; echo -ne "\r$a" ; done

Solution 3

I don' believe there is anything like that. But it should be easy to whip up something along the lines of:

#!/usr/bin/perl

$for_a_while = 1;

$oldcount = -1;
$count = 0;
open($fh, "<", $ARGV[0]);

for (;;) {
  for ($curpos = tell($fh); <$fh>; $curpos = tell($fh)) {
    $count++;
  }
  if($count != $oldcount) {
    print "$count\n";
    $oldcount = $count;
  }
  sleep($for_a_while);
  seek($fh, $curpos, 0);
}

(General idea cribbed from perlfunc(1))

Comments

[towi]

Whow to your tail | awk solution. Know your options: -n +0 would not have occurred to me in this combination.

[Stéphane Chazelas]

The number would increment every time you do a printf foo >> file. You'd need to count the newline characters (like wc -l does in the shell solution I suggested), not the records returned by <$fh>. I don't think you need to use tell or seek at all.

[Stéphane Chazelas]

Try it for yourself, upon reaching end of file, <$fh> will return a record even if it's not terminated by a newline character. So if perl is sitting at the end of the file, and someone later does a printf foo >> file, then <$fh> will return foo (not a line since it's not terminated by a newline character), and $count will be incremented even though no extra line has been added to the file.

[vonbrand]

OP was supposed to monitor logfiles written a line at a time?

[Stéphane Chazelas]

No, which is why your solution may not work. For instance, if the applications writing to the file buffers its output, then at any given time, the last line is likely not to be terminated so will be counted twice.

[towi]

whoo! pv -- another useful new tool. thanks a bunch.

[tombolinux]

With grep you can add a filter to your stream: tail -n +0 -f <my.log> | grep --line-buffered <mystring> | awk '{printf "\r%lu", NR}'

[Stéphane Chazelas]

@tombolinux, awk is a superset of grep. tail -n +0 -f file | awk '/mystring/ {printf "\r%lu", ++n}'

[pLumo]

Cool. I add END{print ""} to make awk print a newline at the end.