Awk - output the second line of a number of .dat files to one file

Question

I have multiple files something like: (in reality i have 80)

file1.dat

2 5

6 9

7 1

file2.dat

3 7

8 4

1 3

I want to end up with a file containing all of the second lines. i.e.

output.dat

6 9

8 4

What i have so far loops though the file names but then over-writes the file before it. e.g. the output of the above files would just be

8 4

my shell script looks like this:

post.sh

TEND = 80

TINDX = 0

while [ $TINDX - lt $TEND]; do

awk '{ print NR==2 "input-$TINDX.dat > output.dat

TINDX = $((TINDX+1))

done

Solution 1

Remove while loop and make use of shell brace expansion and also FNR, a built-in awk variable:

awk 'FNR==2{print $0 > "output.dat"}' file{1..80}.dat

Solution 2

sed would be enough:

sed -sn 2p file{1..80}.dat > output.dat

-s option is needed to print 2nd line from each file, otherwise only 2nd line of first file will be printed.

Solution 3

What about ... head -n 2 input.dat | tail -n 1 | awk ...

Solution 4

aragaer’s sed solution is nicest, yes. But I since I do enjoy a bit of head|tail cutting, have a head|tail solution that supports multiple files, not just a single input.dat. Using a for-loop, instead of passing a list of files to sed, also makes it easier to do other thing with the file before/after extracting the second line with sed.

# empty output.dat first
rm output.dat

# have a one-liner
for file in *.dat; do head -2 $file | tail -1 >> output.dat; done 

Copiously-commented multi-line version:

NB: the code below will run. We are free to put a linebreak after a |, &&, or ||, and continue our command on the next line; we can even put comments in between. I spent years not knowing this (and not really seeing it anywhere). This style is less useful at the interactive prompt, but cleans up script files no end.

# empty output.dat first
rm output.dat

for file in *.dat; do
    # file -> lines 1 and 2 of file
    head -2 $file |
    # lines 1 and 2 of file -> line 2 of file >> appended to output.dat
    tail -1 >> output.dat
done

View more solutions

Comments

[Арсений Черенков]

even shorter awk 'FNR==2' file{1..80}.dat > output.dat

[jimmij]

Yes, head/tail is definitely an option, you don't need awk then.