How do you list number of lines of every file in a directory in human readable format.

Question

I have a list of directories and subdirectories that contain large csv files. There are about 500 million lines in these files, each is a record. I would like to know

1. How many lines are in each file.
2. How many lines are in directory.
3. How many lines in total

Most importantly, I need this in 'human readable format' eg. 12,345,678 rather than 12345678

It would be nice to learn how to do this in 3 ways. Plain vanilla bash tools, awk etc., and perl (or python).

Solution 1

How many lines are in each file.

Use wc, originally for word count, I believe, but it can do lines, words, characters, bytes, and the longest line length. The -l option tells it to count lines.

wc -l <filename>

This will output the number of lines in :

$ wc -l /dir/file.txt
32724 /dir/file.txt

You can also pipe data to wc as well:

$ cat /dir/file.txt | wc -l
32724
$ curl google.com --silent | wc -l
63

How many lines are in directory.

Try:

find . -name '*.pl' | xargs wc -l

another one-liner:

( find ./ -name '*.pl' -print0 | xargs -0 cat ) | wc -l

BTW, wc command counts new lines codes, not lines. When last line in the file does not end with new line code, this will not counted.

You may use grep -c ^ , full example:

#this example prints line count for all found files
total=0
find /path -type f -name "*.php" | while read FILE; do
     #you see use grep instead wc ! for properly counting
     count=$(grep -c ^ < "$FILE")
     echo "$FILE has $count lines"
     let total=total+count #in bash, you can convert this for another shell
done
echo TOTAL LINES COUNTED:  $total

How many lines in total

Not sure that I understood you request correctly. e.g. this will output results in the following format, showing the number of lines for each file:

# wc -l `find /path/to/directory/ -type f`
 103 /dir/a.php
 378 /dir/b/c.xml
 132 /dir/d/e.xml
 613 total

Alternatively, to output just the total number of new line characters without the file by file counts to following command can prove useful:

# find /path/to/directory/ -type f -exec wc -l {} \; | awk '{total += $1} END{print total}'
 613

Most importantly, I need this in 'human readable format' eg. 12,345,678 rather than 12345678

Bash has a printf function built in:

printf "%0.2f\n" $T

As always, there are many different methods that could be used to achieve the same results mentioned here.

Solution 2

In many cases combining the wc command and the wildcard * may be enough.
If all your files are in a single directory you can call:

wc -l src/*

You can also list several files and directories:

wc -l file.txt readme src/* include/*

This command will show a list of the files and their number of lines.
The last line will be the sum of the lines from all files.

---

To count all files in a directory recursively:

First, enable globstar by adding shopt -s globstar to your .bash_profile. Support for globstar requires Bash ≥ 4.x which can be installed with brew install bash if needed. You can check your version with bash --version.

Then run:

wc -l **/*

Note that this output will be incorrect if globstar is not enabled.

Solution 3

This command will give list of lines code in each directory:

find . -name '*.*' -type f | xargs wc -l

Solution 4

I will just augment @malyy answer for the following (to big for a comment):

How many lines in total

Many answers are using wc command line file option with xargs. The problem with this is xargs is limited to a rather small platform dependent size.

Furthermore there is a difference between BSD (macOS) and GNU (linux/homebrew) wc.

The GNU one is ideal because it can read the file listing from a file instead of arguments (--files0).

If you are on mac and have homebrew you should do the following:

find . -name "*.pl" -print0 | gwc -l --files0=-

Notice the gwc instead of wc.

Solution 5

a bit late to the game, but I got a bunch of argument errors with the above due to the size of the dir. This worked for me:

for i in $(find . -type f); do wc -l $i; done >> /home/counts.txt

View more solutions

Comments

[Hexatonic]

By the way, how do I use printf in your examples? I tried to pipe to it from wc -l, but it didn't work.

[malyy]

try > find . -name '*.pl' | xargs wc -l | awk '{printf ("%0.2f ", $1) } {print $2}' change the output of 'printf' for your needs

[Hexatonic]

This doesn't add commas to the number to make it more human readable though. It just adds a zeros to the end.

[Hexatonic]

echo 1000000000000 | xargs printf "%'d\n" 1,000,000,000,000

[Taylor D. Edmiston]

And for counting files in the currrent directory recursively: wc -l **/*

[BallpointBen]

@Hexatonic printf doesn't read its arguments from stdin, but rather from the command line (compare piping to echo vs piping to cat; cat reads from stdin, echo doesn't). Instead, use printf "$(find ... | xargs ...)" to supply the output as arguments to printf.

[M. Justin]

@TaylorEdmiston For me (on Mac) that only counts the files exactly one directory down. It skips the files in the current directory, and for any instance that would be more than one directory deep it warns that it's a directory: "wc: parent_dir/child_dir: read: Is a directory"

[Taylor D. Edmiston]

@Thomio It requires globstar to be enabled. On macOS, I believe it is disabled out of the box. I've just sent an edit to your answer that adds the command and how to enable globstar.

[Adam Gent]

n.b. xargs is limited to a certain size and thus you can get spurious results. Unfortunately only the GNU wc command takes a stream/file input for file listings. See my answer.

[Bluebaron]

That is a quality answer.

[Aaron Franke]

This doesn't seem to work for files with spaces in the file name.