How to cat all the log files within a range of dates

Question

I have log files which are named in the following manner:

localhost_log_file.2017-09-01.txt
localhost_log_file.2017-09-02.txt
....

localhost_log_file.2017-10-30.txt

In other words, each log file has the following form:

localhost_log_file.YYYY-MM-DD.txt

I want to cat all the log files taken between the dates of 2017-09-03 and 2017-10-08, i.e. every log file starting from localhost_log_file.2017-09-03.txt through localhost_log_file.2017-10-08.txt.

Currently what I do is produce three intermediate files by separately executing each of the following commands:

for((i=3;i<=9;i++)) do cat localhost_log_file.2017-09-0$i.txt >> log1.csv ; done;

for((i=10;i<=30;i++)) do cat localhost_log_file.2017-09-$i.txt >> log2.csv ; done;

for((i=1;i<=8;i++)) do cat localhost_log_file.2017-10-0$i.txt >> log3.csv ; done;

Then I combine the intermediate files as follows:

cat log1.csv log2.csv log3.csv> totallog.csv

Is there a better way to do this?

Solution 1

How about using the date utility to iterate through the range of dates you're interested in? Here is what that might look like for your example:

# Set the date counter to the start date
d=2017-09-03

# Iterate until we reach the end date (i.e. the date after the last date we want)
while [ "$d" != 2017-10-09 ]; do

    # cat each file
    cat "localhost_log_file.${d}.txt";

    # Increment the date counter
    d="$(date -I -d "$d + 1 day")";

done

See this for more information:

• Bash: Looping through dates

Alternatively, you can pass the results of the loop to the cat command instead of invoking cat in the body of the loop.

Here is what that could look like using command-substitution:

d=2017-09-03
cat $(while [ "$d" != 2017-10-09 ]; do
    echo "localhost_log_file.${d}.txt";
    d="$(date -I -d "$d + 1 day")";
done)

And here is the same thing using a pipe and xargs:

d=2017-09-03
while [ "$d" != 2017-10-09 ]; do
    echo "localhost_log_file.${d}.txt";
    d="$(date -I -d "$d + 1 day")";
done | xargs cat

Solution 2

You can nest brace expansion.

Short and sweet:

cat localhost_log_file.2017-{09-{03..30},10-{01..08}}.txt > totallog.csv

---

Note that some systems such as macOS use an older version of Bash where this doesn't work, as the leading zeros are stripped from brace expansion integer sequences. For Linux this works fine.

Solution 3

use date:

for i in {0..35}; do
     cat localhost_log_file.$(date +%F -d "2017-09-03 + $i day").txt
done > totallog.csv

Solution 4

Better is subjective, but you can use brace expansion for both

cat localhost_log_file.2017-09-{03..09}.txt > log1.csv

and

cat log{1..3} > totallog.csv

Assuming you wanted to keep the log?.csv files

View more solutions

Comments

[Wildcard]

Try echo {03..09} and see what you get. The leading zeros are not preserved.

[bu5hman]

Already checked on my machine, worked 100% in my bash shell with cat log{02..10}.csv and so does your echo......

[Wildcard]

Aha, right you are. Now I've undeleted my answer and added the caveat about bash versions.

[bu5hman]

I like your solution but I only assumed that the OP split the month of 09 because he wanted to keep the data separate. Other than that brace expansion is more elegant.

[igal]

Small comment: Using brace expansion work really for this specific task (and is very concise!), but using the date function generalizes better and is much more flexible, e.g. it will automatically handle varying month lengths, leap years, etc.