How can I achieve portability with sed -i (in-place editing)?

shell sed freebsd gnu portability

18,037

Solution 1

GNU sed accepts an optional extension after -i. The extension must be in the same argument with no intervening space. This syntax also works on FreeBSD sed.

sed -i.bak -e '…' SOMEFILE

Note that on FreeBSD, -i also changes the behavior when there are multiple input files: they are processed independently (so e.g. $ matches the last line of each file). Also this won't work on BusyBox.

If you don't want to use backup files, you could check which version of sed is available.

# Assume that sed is either FreeBSD/macOS or GNU
case $(sed --help 2>&1) in
  *GNU*) set sed -i;;
  *) set sed -i '';;
esac
"$@" -e '…' "$file"

Or alternatively, to avoid clobbering the positional parameters, define a function.

case $(sed --help 2>&1) in
  *GNU*) sed_i () { sed -i "$@"; };;
  *) sed_i () { sed -i '' "$@"; };;
esac
sed_i -e '…' "$file"

If you don't want to bother, use Perl.

perl -i -pe '…' "$file"

If you want to write a portable script, don't use -i — it isn't in POSIX. Do manually what sed does under the hood — it's only one more line of code.

sed -e '…' "$file" >"$file.new"
mv -- "$file.new" "$file"

Solution 2

If you don't find a trick to make sed play nice, you could try:

Don't use -i :

sed '1s/^/<!DOCTYPE html> \n/' "${file_name.html}" > "${file_name.html}.tmp" &&
  mv "${file_name.html}.tmp" "${file_name.html}"

Use Perl

perl -i -pe 'print "<!DOCTYPE html> \n" if $.==1;' "${file_name.html}"

Solution 3

ed

You can always use ed to prepend a line to an existing file.

$ printf '0a\n<!DOCTYPE html>\n.\nw\n' | ed my.html

Details

The bits around the <!DOCTYPE html> are commands to ed instructing it to add that line to the file my.html.

sed

I believe this command in sed can also be used:

$ sed -i '1i<!DOCTYPE html>\n` testfile.csv

Solution 4

You can also do manually what perl -i does under the hood:

{ rm -f file && { echo '<!DOCTYPE html>'; cat; } > file;} < file

Like perl -i, there's no backup, and like most solutions given here, beware it may affect the permissions, ownership of the file and may turn a symlink into a regular file.

With:

sed '1i\
<!DOCTYPE html>' file 1<> file

sed would overwrite the file over itself, so would not affect ownership and permissions or symlinks. It works with GNU sed because sed will typically have read a buffer full of data from file (4k in my case) before overwriting it with the i command. That wouldn't work if the file was more than 4k except for the fact that sed also buffers its output.

Basically sed works on blocks of 4k for reading and writing. If the line to insert is smaller than 4k, sed will never overwrite a block it has not read yet.

I wouldn't count on it though.

Solution 5

FreeBSD sed, which is used on Mac OS X as well, needs the -e option after the -i switch to define & recognise the following (regex) command correctly & unambiguously.

In other words, sed -i -e ... should work with both FreeBSD & GNU sed.

More generally, omitting the backup extension after FreeBSD sed -i requires some explicit sed option or switch following the -i to avoid confusion on part of FreeBSD sed while parsing its command-line arguments.

(Note, however, that sed in-place file edits lead to file inode changes, see "In-place" editing of files).

(As a general hint, recent versions of FreeBSD sed have the -r switch to increase compatibility with GNU sed).

echo a > testfile.txt
ls -li testfile.txt
#gsed -i -e 's/a/A/' testfile.txt
#bsdsed -i 's/a/A/' testfile.txt  # does not work
bsdsed -i -e 's/a/A/' testfile.txt
ls -li testfile.txt
cat testfile.txt

View more solutions

18,037

Red

Updated on September 18, 2022

Comments

Red over 1 year
I'm writing shell scripts for my server, which is a shared hosting running FreeBSD. I also want to be able to test them locally, on my PC running Linux. Hence, I'm trying to write them in a portable way, but with sed I see no way to do that.

Part of my website uses generated static HTML files, and this sed line inserts correct DOCTYPE after each regeneration:
```
sed -i '1s/^/<!DOCTYPE html> \n/' ${file_name.html}
```
It works with GNU sed on Linux, but FreeBSD sed expects the first argument after -i option to be extension for backup copy. This is how it would look like:
```
sed -i '' '1s/^/<!DOCTYPE html> \n/' ${file_name.html}
```
However, GNU sed in turn expects the expression to follow immediately after -i. (It also requires fixes with newline handling, but that's already answered in here)

Of course I can include this change in my server copy of the script, but that would mess i.e. my use of VCS for versioning. Is there a way to achieve this with sed in a fully portable way?
- Mathias Begert over 10 years
  
  The two sed snippets you provided are identical, are you sure there isn't a typo? Also, i am able to execute GNU sed supplying the backup extension right after -i
- Red over 10 years
  
  Duh, thanks for spotting this. I've fixed my question. The second line results in error in my sed, it expects '1s/^/<!DOCTYPE html> \n/' to be a file and complains it can't find it.
- MvG over 7 years
  
  Cross reference: sed in-place flag that works both on Mac (BSD) and Linux on Stack Overflow.
Red over 10 years

I eventually resorted to Perl, but using ed is a good alternative that's not popular among Unix-like users as it should be.
slm over 10 years

@Red - glad to hear you resolved you issue. Yeah I'd not seen that one before, googling turned it up and it actually seemed like the most portable, apt way to do this.
mikeserv over 9 years

GNU sed -i also implies -s. And the easiest way to check for a GNU sed is with the sed v command which is a valid noop for GNU but fails everywhere else.
Ivan X about 9 years

Inspired by the above tips, here's an single-line (if ugly) portable version for those who really want one, though it does spawn a subshell: sed -i$(sed v < /dev/null 2> /dev/null || echo -n " ''") -e '...' "$file" If it's not GNU sed, it inserts a space followed by two singe quotes after -i so that it works on BSD. GNU sed gets only -i.
Gilles 'SO- stop being evil' about 9 years

@IvanX I'm wary of using the presence of the v command to test for GNU sed. What if FreeBSD decided to implement it?
Ivan X about 9 years

@Gilles Fair point, but the man page for GNU sed describes v as being exactly for that purpose (detecting that it's GNU sed and not something else), so one would hope that *BSD would honor that. I can't think of another test, offhand, that takes no action on GNU sed, while causing an error on BSD sed (or vice versa), other than using -i itself, but that would require creating a dummy file first. Your test for sed above is OK but unwieldy for inline. Avoiding -i entirely, as you suggest, certainly seems like the safest bet, but I'm ok with using sed v given that's its purpose for existing.
Stéphane Chazelas almost 9 years

No, bsdsed -i -e 's/a/A/' is not in-place editing, it's editing with saving the original with a "-e" suffix (testfile.txt-e).
Stéphane Chazelas almost 9 years

note that GNU sed also supports -E (in addition to -r) for compatibility with FreeBSD. -E is likely to be specified in the next POSIX version, so we should all be forgetting about that -r non-sense and pretend it never existed.
A.D. almost 9 years

Should be echo '<!DOCTYPE html>' or escaped without "" quotes.
Stéphane Chazelas almost 9 years

@A.D. Good point. I tend to forget about that bug^Wfeature of interactive bash/zsh as I generally disable it for myself.
Wildcard about 8 years

This is one of the very few answers here that doesn't have a wide open security hole of redirecting to a "temp file" with a static, predictable name without checking if it already exists. I believe this should be the accepted answer. Very nice demonstration of the use of group commands, also.
Hallaghan about 8 years

The trick with "$@" is very nice, but I wouldn't use it in a longer script. Unfortunately normal variables are not expanded in that way (and sh doesn't have arrays) so I couldn't find anything better.
Gilles 'SO- stop being evil' about 8 years

@lapo An alternative is to define a function, see my edit.
Wildcard over 7 years

There's nothing Vim-specific here. This is fully compliant with POSIX specifications for ex, except that implementations are not required to support multiple -c flags. For definite portability I would use printf '%s\n' 1i '<!DOCTYPE html>' . x | ex file
someonewithpc about 3 years

Not sure when it changed, but busybox 1.32 supports '-i.bak' just fine
laubster about 3 years

I really like this. As to losing permissions (which perl -i preserves), one could stat -c %a before the rm and then chmod before the close brace. But not only are you then getting into ugly code territory, you'd also need to consider that a setgid bit on the original file may not be desirable on the new one since the group may have changed.
Admin almost 2 years

It's not GNU vs BSD, it's GNU/NetBSD/OpenBSD/busybox/toybox vs FreeBSD/macos.