Command to display first few and last few lines of a file

65,543

Solution 1

You can use sed or awk to make it with one command. However you'll loose at speed, cause sed and awk will need to run through the whole file anyway. From a speed point of view it's much better to make a function or every time to combination of tail + head. This does have the downside of not working if the input is a pipe, however you can use proccess substitution, in case your shell supports it (look at example below).

first_last () {
    head -n 10 -- "$1"
    tail -n 10 -- "$1"
}

and just launch it as

first_last "/path/to/file_to_process"

to proceed with process substitution (bash, zsh, ksh like shells only):

first_last <( command )

ps. you can even add a grep to check if your "global conditions" exist.

Solution 2

@rush is right about using head + tail being more efficient for large files, but for small files (< 20 lines), some lines may be output twice.

{ head; tail;} < /path/to/file

would be equally efficient, but wouldn't have the problem above.

Solution 3

The { head; tail; } solution wouldn't work on pipes (or sockets or any other non-seekable files) because head could consume too much data as it reads by blocks and can't seek back on a pipe potentially leaving the cursor inside the file beyond what tail is meant to select.

So, you could use a tool that reads one character at a time like the shell's read (here using a function that takes the number of head lines and tail lines as arguments).

head_tail() {
  n=0
  while [ "$n" -lt "$1" ]; do
    IFS= read -r line || { printf %s "$line"; break; }
    printf '%s\n' "$line"
    n=$(($n + 1))
  done
  tail -n "${2-$1}"
}
seq 100 | head_tail 5 10
seq 20 | head_tail 5

or implement tail in awk for instance as:

head_tail() {
  awk -v h="$1" -v t="${2-$1}" '
    {l[NR%t]=$0}
    NR<=h
    END{
      n=NR-t+1
      if(n <= h) n = h+1
      for (;n<=NR;n++) print l[n%t]
    }'
}

With sed:

head_tail() {
  sed -e "1,${1}b" -e :1 -e "$(($1+${2-$1})),\$!{N;b1" -e '}' -e 'N;D'
}

(though beware that some sed implementations have a low limitation on the size of their pattern space, so would fail for big values of the number of tail lines).

Solution 4

With the -u (--unbuffered) option of GNU sed, you can use sed -u 2q as an unbuffered alternative to head -n2:

$ seq 100|(sed -u 2q;tail -n2)
1
2
99
100

When (head -n2;tail -n2) is used in a pipeline, it doesn't print the last lines when the last lines are part of a block of input that is consumed by head:

$ seq 1000|(head -n2;tail -n2)
1
2
999
1000
$ seq 100|(head -n2;tail -n2)
1
2

If the input is a regular file and not a pipeline, you can just use (head -n2;tail -n2):

$ seq 100 >a;(head -n2;tail -n2)<a
1
2
99
100

Solution 5

Using bash process substitution, you can do the following:

make_some_output | tee >(tail -n 2) >(head -n 2; cat >/dev/null) >/dev/null

Note that the lines are not guaranteed to be in order, though for files longer than about 8kB, they very likely will be. This 8kB cutoff is the typical size of the read buffer, and is related to the reason | {head; tail;} doesn't work for small files.

The cat >/dev/null is necessary to keep the head pipeline alive. Otherwise tee will quit early, and while you'll get output from tail, it'll be from somewhere in the middle of the input, rather than the end.

Finally, why the >/dev/null instead of, say, moving tail to another |? In the following case:

make_some_output | tee >(head -n 2; cat >/dev/null) | tail -n 2  # doesn't work

head's stdout is fed into the pipe to tail rather than the console, which isn't what we want at all.

Share:
65,543

Related videos on Youtube

ekoeppen
Author by

ekoeppen

Updated on September 18, 2022

Comments

  • ekoeppen
    ekoeppen over 1 year

    I have a file with many rows, and each row has a timestamp at the starting, like

    [Thread-3] (21/09/12 06:17:38:672) logged message from code.....
    

    So, I frequently check 2 things from this log file.

    1. First few rows, that has the global conditions and start time is also given.
    2. Last few rows, that has the exit status with some other info.

    Is there any quick handy single command that could let me display just the first and last few lines of a file?

    • Admin
      Admin over 11 years
      What's global conditions, and doesn't head and tail works for you?
    • Admin
      Admin over 11 years
      That is the part of my log file. I was trying to be elaborative. You can ignore that.
    • Admin
      Admin over 11 years
      Your solution looks fine to me. If you want more convenience, make it into a shell function (even an alias might do).
    • Admin
      Admin over 11 years
      @vonbrand Problem is that I don't know N
    • Admin
      Admin over 11 years
      @Bernhard, I'm no sed(1) expert, but there are ways of stashing stuff away for later use with it. Maybe it pays off to look in there. OTOH, I'd probably whip up a Perl (or whatever) script to do it if used frequently, as I'm more familiar with that.
    • Admin
      Admin over 11 years
      One thing is that it doesn't need to be convenient and easy to remember, you just have to put it in your .bashrc, or as a script in $HOME/bin, etc.
    • Admin
      Admin over 11 years
      If the input is a pipe, see unix.stackexchange.com/questions/66408/…
  • Marco
    Marco over 11 years
    In contrast to rushs solution, this does not work in a POSIX shell.
  • Gilles 'SO- stop being evil'
    Gilles 'SO- stop being evil' over 11 years
    @Marco Huh? Only POSIX constructs are used here. What do you see going wrong?
  • Marco
    Marco over 11 years
    @Gilles I missed the space: {head; tail;} < file works in zsh but fails in sh. { head; tail;} < file always works. Sorry for the noise.
  • derobert
    derobert over 11 years
    When head or tail finish writing the output they want, they close their stdin and exit. That's where the SIGPIPE is coming from. Normally this is a good thing, they're discarding the rest of the output, so there is no reason for the other side of the pipe to continue spending time generating it.
  • Gilles 'SO- stop being evil'
    Gilles 'SO- stop being evil' over 11 years
    What makes the order likely to be upheld? It probably will be for a large file, because tail has to work longer, but I expect (and do see) it failing about half the time for short inputs.
  • Stéphane Chazelas
    Stéphane Chazelas over 11 years
    You'll get the SIGPIPE with tee >(head) >(tail) for the same reasons (>(...) which by the way is a ksh feature now supported by both zsh and bash as well) uses pipes as well. You could do ... | (trap '' PIPE; tee >(head) >(tail) > /dev/null) but you'll still see some broken pipes error messages from tee.
  • Jander
    Jander over 11 years
    On my system (bash 4.2.37, coreutils 8.13), tail is the one being killed by SIGPIPE, not tee, and tail isn't writing to a pipe. So it must be from a kill(), right?. And this only happens when I'm using the | syntax. strace says that tee isn't calling kill()... so maybe bash?
  • Jander
    Jander over 11 years
    @Stephane - I've never seen the SIGPIPE with the >(head) >(tail) syntax. I always do with the | syntax.
  • Stéphane Chazelas
    Stéphane Chazelas over 11 years
    @Jander, try feeding more than 8k like seq 100000 | tee >(head -n1) >(tail -n1) > /dev/null
  • Jander
    Jander over 11 years
    @Stephane -- Ah, in that case tee quits early, SIGPIPE is sent to seq, and tail prints what from its view is the last line of the stream... in my case, 2679. But I finally understand why tail was getting a SIGPIPE... its output was being fed into head along with tee's. I've updated my answer -- thanks a lot for the debugging help!
  • Stéphane Chazelas
    Stéphane Chazelas over 11 years
    Note that with seq 3 | ..., line 2 will be output twice which may or may not be desirable, I've taken special care to avoid it in the solution I gave.
  • l0b0
    l0b0 over 11 years
    -n 10 is the default, no?
  • rush
    rush over 11 years
    @l0b0 yes, it's default. -n 10 is not necessary here.
  • Atcold
    Atcold about 10 years
    For multiple files we for f in *; do { head; tail;} < $f; done
  • FCTW
    FCTW almost 10 years
    How would you run this with sudo?
  • don_crissti
    don_crissti over 9 years
    Shorter: ed -s file <<< $'11,$-10d\n,p\nq\n'
  • Stéphane Chazelas
    Stéphane Chazelas over 8 years
    @FCTW, sudo sh -c '{ head; tail;} < /path/to/file'
  • FCTW
    FCTW over 8 years
    @StéphaneChazelas TYVM! Long ago as I asked this, I've probably learned enough at my job to have figured this out, but your response is FANTASTIC! :-)
  • Ben Usman
    Ben Usman almost 7 years
    this should be the top answer! works like a charm!
  • nisetama
    nisetama about 3 years
    This is basically the same as sed -n '1,3p;$p'.