How do I trim bytes from the beginning and end of a file?

Question

I have a file, that has trash (binary header and footer) at the beginning and end of the file. I would like to know how to nuke these bytes. For an example, let's assume 25 bytes from the beginning. And, 2 bytes from the end.

I know I can use truncate and dd, but truncate doesn't work with a stream and it seems kind of cludgey to run two commands on the hard file. It would be nicer if truncate, knowing how big the file was, could cat the file to dd. Or, if there was a nicer way to do this?

Solution 1

You can combine GNU tail and head:

tail -c +26 file | head -c -2

will output the contents of file starting at byte 26, and stopping two bytes (minus two -2) before the end. (-c operates on bytes, not characters.)

Solution 2

dd will do both for you in a single command. Set the block size to 1 byte, skip the 25 first bytes, count to the size of file minus skip and end bytes.

100 byte file
file.img

dd if=./file.img of=./trimed_file.img bs=1 skip=25 count=73

Double check numbers cause it might count from 0.

Solution 3

With ksh93:

{ head -c "$n"; } < file <#((n = EOF - 25 - 2 , 25))

Or to do it in-place:

{ head -c "$n"; } < file <#((n = EOF - 25 - 2 , 25)) 1<>; file

If you have /opt/ast/bin ahead of your $PATH, you'll get the head builtin.

• <#((...)) is a lseek() operator. ... is interpreted as an arithmetic expression where EOF is the length of the file. So above, we're assigning the length of the portion to display to $n and seeking to 25 bytes within the file.
• <>; is a open-in-read+write-mode-and-truncate-if-the-command-is-successful redirection operator.

Comments

[Ronit]

isn't too slow?

[jc__]

Too slow? Only if the file is very large like many GB. Then you could adjust the block size.

[Ronit]

it took 0m8.319s for a 1.3 MB file, For the count he can use something like $(( stat -c %s test` - 5 ))` I guess.

[jc__]

Remember that skip and count are blocks not bytes. Pick the largest multiple for the block size. The closer the block size is to the HD buffer size the more efficient and faster the command is.

[Ronit]

Yeah, however bs is defined 1 in your command :-) so in this case it will work...

[ilkkachu]

with bs=1 dd makes two system calls for each byte, that's horribly slow. GNU dd has the count_bytes and skip_bytes that can help, do something like dd bs=8192 iflag=count_bytes,skip_bytes skip=25 count=73 < infile > outfile

[Stéphane Chazelas]

Note that tail -c +26 is standard (head -c -2 is not).

[cat]

@StéphaneChazelas is there a POSIX compliant version of the head -c -2 command?

[Zorglub29]

This does not work well when the file contains null bytes, right?

[Stephen Kitt]

@Zorglub29 it works for me even with null bytes.