Cannot sum numbers received from stdin using bc

Question

I'm trying to calculate the average entropy of files contained in a folder using:

{ echo '('; find . -type f -exec entropy {} \; | \
  grep -Eo '[0-9.]+$' | \
  sed -r 's/$/+/g'; echo '0)/'; 
  find . -type f | wc -l; }  | \
tr -d '\n' | bc -l

entropy being an executable which calculates the Shannon entropy of a file, giving outputs of the form:

$ entropy foo
foo: 5.13232

The aforementioned command errors out with:

(standard_in) 1: syntax error

However, the generated output seems to have no problems:

$ { echo '('; find . -type f -exec entropy {} \; | \
    grep -Eo '[0-9.]+$' | \
    sed -r 's/$/+/g'; echo '0)/'; \
    find . -type f | wc -l; }  | \
  tr -d '\n'
(5.13232+2.479+1.4311+0)/3

And this works too:

$ echo '(2.1+2.1)/2' | bc -l
2.1

What is wrong with the mentioned command?

Solution 1

And this works too: echo '(2.1+2.1)/2' | bc -l

Ah, but did you try:

echo '(2.1+2.1)/2' | tr -d '\n' | bc -l
(standard_in) 1: syntax error

Using echo -n will accomplish the same thing -- there's no terminating newline, and that's your problem.

Solution 2

bc has some pretty particular syntax. dc is less choosy:

find . -type f -exec entropy \{\} + |
sed 's/.*://;N;N;s/\n[^:]*:/+/g;s/+//;s|$| 3/p|' |
dc

I think that does what you're trying to do, but I'm not totally certain. An output sample larger than a single line would help.

Comments

[Bernhard]

Are you willing to use awk? Would be substantially easier.

[gniourf_gniourf]

You're just missing a trailing endline for the bc command: compare printf '(5.13232+2.479+1.4311+0)/3' | bc -l with echo '(5.13232+2.479+1.4311+0)/3' | bc -l. (your tr -d '\n' command removes the trailing newline that bc needs).

[gniourf_gniourf]

An easy fix is to insert { cat; echo; } between the tr and the bc: tr -d '\n' | { cat; echo; } | bc -l or to replace the tr -d '\n' part with: { tr -d '\n'; echo; }

[Stéphane Chazelas]

Use paste -sd'\0' - instead of tr -d '\n' to preserve the last newline character. (see also paste -sd+ - to join lines with +).

[gniourf_gniourf]

You could use dc to do all the work too: { find . -type f -exec entropy \{\} | sed 's/.*://' ; echo ' 10k[+]sa[z2!>az2!>b]sbzsclbxlc/p'; } | dc. The ugly 10k[+]sa[z2!>az2!>b]sbzsclbxlc/p is a bunch of junk that tells dc to compute the average of the numbers left on the stack (with a scale of 10) :D.

[mikeserv]

@gniourf_gniourf - it's pretty crazy fast, though. I wanna learn how to use it better. Thanks for the study material...