Delete the first n bytes of files
Solution 1
for file in /foo/*
do
if [ -f "$file" ]
then
dd if="$file" of="$file.truncated" bs=31 skip=1 && mv "$file.truncated" "$file"
fi
done
or the faster, thanks to Gilles' suggestion:
for file in /foo/*
do
if [ -f $file ]
then
tail +32c $file > $file.truncated && mv $file.truncated $file
fi
done
Note: Posix tail specify "-c +32" instead of "+32c" but Solaris default tail doesn't like it:
$ /usr/bin/tail -c +32 /tmp/foo > /tmp/foo1
tail: cannot open input
/usr/xpg4/bin/tail
is fine with both syntaxes.
If you want to keep the original file permissions, replace
... && mv "$file.truncated" "$file"
by
... && cat "$file.truncated" "$file" && rm "$file.truncated"
Solution 2
The following commands cut first 31 bytes from $file
(using $file~
as a temp. copy):
dd if="$file" of="$file~" bs=1 skip=31
mv "$file~" "$file"
You only need to list or find
all files under /foo/
and execute the two above for every $file
found.
Solution 3
tail -c +32
outputs its input minus the first 31 bytes. (Yes, the argument is off by one.) To edit a file in place, use sponge in a loop, or if you don't have it and don't want to bother, do its job in the shell:
for x in /foo/*; do tail -c +32 "$x" | sponge "$x"; done
for x in /foo/*; do tail -c +32 "$x" >"$x.new" && mv "$x.new" "$x"; done
If the commands are interrupted for whatever reason (e.g. a power failure), it might be hard to figure out where you left off. Writing the new files to a separate directory would make things easier.
mkdir /foo.tmp
cd /foo
for x in *; do tail -c +42 -- "$x" >"/foo.tmp/$x" && rm -- "$x"; done
mv /foo.tmp/* /foo
rmdir /foo.tmp
If the files are really large (as in, large enough that having two copies of even a single one is a problem), you can use one of the techniques mentioned in this thread.
Solution 4
You can use Vim in Ex mode:
for each in /foo/*
do
ex -sc '%!tail -c+32' -cx "$each"
done
%
select all lines!
run commandx
save and close
Related videos on Youtube
von der tann
Updated on September 18, 2022Comments
-
von der tann over 1 year
I've got an extreme problem, and all of the solutions I can imagine are complicated. According to my UNIX/Linux experience there must be an easy way.
I want to delete the first 31 bytes of each file in
/foo/
. Each file is long enough. Well, I'm sure somebody will deliver me a suprisingly easy solution I just can't imagine. Maybe awk?-
Angel Todorov almost 13 yearsAny awk/sed/ed solution will be line-oriented, so if you don't know the first line will be at least 31 characters then complications ensue.
-
-
jlliagre almost 13 yearsSwapping bs and skip values will increase the performance.
-
Gilles 'SO- stop being evil' almost 13 yearsSuggesting
dd
here is overkill,tail
is more appropriate (simpler, less risk of a killer typo, no spurious messages on stderr). -
jlliagre almost 13 yearsYou are right. I usually avoid commands intended to process text files when processing possibly binary ones but "tail +32c" will work here.
-
Peter.O almost 13 years@jlliagre: You have written
cut
(shouldn't that be tail? ... asis, it doesn't work for me... -
jlliagre almost 13 yearsOf course, it's tail. Sorry about the mismatch.
-
Gilles 'SO- stop being evil' almost 13 years@jlliagre: On Solaris, you should have
/usr/xpg4/bin
ahead of/usr/bin
on yourPATH
, or you'll be stuck in the early 1990s. Many unices (e.g. GNU, BusyBox) no longer support the historical+32c
syntax, and take it to mean a file called+32c
(as POSIX requires). -
jlliagre almost 13 years@Gilles: I should indeed as I usually strongly support standard conformance but in that specific case, I'm afraid I won't switch soon. What refrains me to do it is that change would trigger a gazillion of issues with existing scripts, and would distance me from the community of Solaris users who in their very large majority are still using the traditional commands.
-
Aryeh Armon almost 7 yearstail: cannot open '+32c' for reading: No such file or directory
-
jlliagre almost 7 years@AryehArmon As stated in my reply, you can use instead "tail -c +32 $file" on POSIX systems.
-
Aryeh Armon almost 7 years@jlliagre this will give me last 32 bytes, correct?
-
jlliagre almost 7 years@AryehArmon Incorrect, unless if the file size is exactly 64 bytes.
-
Hameda169 over 3 years
mv
changes the file permissions. you can usecat
instead ofmv
like this:tail +32c $file > $file.truncated && cat $file.truncated > $file && rm $file.truncated
@jlliagre