Resources for portable shell programming

Question

What resources exist for portable shell programming? The ultimate answer is to test on all targeted platforms, but that's rarely practical.

The POSIX / Single UNIX specification is a start, but it tells neither you what the level of support of each implementation is, nor what common extensions exist. You can read the documentation of each implementation, but that's very time consuming and not completely accurate.

I seems to me that an ideal format would be some kind of community-annotated version of the POSIX spec, where each feature is annotated by its support level amongst the different implementations. Is there such a thing? Or are there other useful resources?

_{For example, there is Sven Mascheck's shell portability pages, but it's only about syntactic elements and a few built-ins, and only covers old shells. I'm looking for a more comprehensive resource.}

Solution 1

The autoconf manual has a section on portable shell programming.

Although that's not specifically targeting POSIX, it's probably the most complete collection of what to do and not to do when attempting to write portable shell code.

Solution 2

In addition to dash and posh, there's bournesh (or bsh), the Heirloom Bourne Shell, that can be used to detect Bashisms.

The Heirloom Project also includes "The Heirloom Toolchest", a collection of more than 100 standard Unix utilities (which could serve as a starting point for comparing command line options).

Solution 3

Similar to this answer, try executing your script in posh.

Also, don't forget to set the POSIXLY_CORRECT environment variable to true, as this causes many programs (not only the shell) to adhere more strictly to the POSIX standards.

Solution 4

Writing your scripts using dash might be a start.

Solution 5

Today, you can usually find a POSIX shell on a system, and so that generally means you can script in the POSIX language (modulo running into compliance bugs).

The only problem is that /bin/sh is sometimes not a POSIX shell. And you must hard-code the #! line into scripts that are to behave as nice executables; you can't just ask the user to research the problem and then invoke your script as /path/to/posix/shell myscript.

So, the trick is to use POSIX features in your script, but make the script automatically find the POSIX shell. One way to do it is like this:

#!/bin/sh

# At this point, we may be running under some old shell
# we have to tread carefully.

# note how we use test rather than [ ] syntax and avoid
# depending on test with no argument producing a failure;
# i.e. "test $posix_shell".

if ! test x$posix_shell = x ; then
  # the three possible shell paths are just an example;
  # please extend as necessary.

  for shell in /usr/xpg4/bin/sh /bin/bash /usr/bin/bash ; do
    if test -x $shell ; then
       posix_shell=$shell
    fi
  done
  if test x$posix_shell = x ; then
    echo "no POSIX shell found"
    exit 1
    # or we could avoid bailing here and just fall back on /bin/sh:
    # echo "falling back on /bin/sh: cross your fingers that it works"
    # posix_shell=/bin/sh
  fi
  export posix_shell

  # plain "$@" is broken in ancient shells! 
  # I seem to recall ${@+"$@"}: not sure if that's the right trick.

  exec $posix_shell $0 ${@+"$@"}  # can we count on exec in legacy shells? 
fi

# phew, at this point in the script we have been re-executed and are
# being interpreted by some reasonably modern shell. We can use $(...)
# command substitution, and other features.

There are other approaches, such as code generation. Boostrap your scripts with a small script that takes a body of script files without a #! line, and adds one.

The worst possible thing you can do is to start writing entire scripts in such a way that they run on a Bourne shell from 1981. This is only necessary if you must write for a system that doesn't really doesn't have any other shell.

View more solutions

Comments

[Gilles 'SO- stop being evil']

N.B. Please don't answer here just to cite one particular implementation's conformance document. If you have one of those, the corresponding tag wiki would be a good place.

[geekosaur]

Someone needs to mine the revision history for autoconf and Metaconfig (Perl, rn) and collect all the tricks and their explanatory comments in one place.

[Gilles 'SO- stop being evil']

@geekosaur: Good suggestion. I've never looked at the internals of autoconf, is there something one could use as a guide when writing one's own scripts? If so, this would be a good answer to my question, even if it's not a definitive one.

[geekosaur]

There's quite a lot of decently-commented portability stuff in there. It's worth sitting down and reading through one. (Although sometimes older comments get shortened or removed completely, which is why it'd be necessary to check the revision history too.)

[geekosaur]

I should also mention that, while autoconf is somewhat dry reading, Metaconfig's commentary is often quite amusing.

[Mikel]

I find the bigger issue is which non-POSIX extensions are available in which shells. These days every common shell (dash, bash, zsh) should be very close to POSIX conformant, AFAIK.

[Gilles 'SO- stop being evil']

Another related resource is GNUlib, whose documentation lists many portability issues, but for the C API.

[J. Taylor]

I'm sure you both could find it easily, but for the benefit of other people who are following this thread, here's a link to a relevant page from the autoconf manual: gnu.org/software/hello/manual/autoconf/Portable-Shell.html

[Gilles 'SO- stop being evil']

Related: How can I test for POSIX compliance for shell scripts?

[把友情留在无盐]

notice we often write shell scripts combined with external utilities e.g. grep , we may want to pay attention to portability of those utilities alongside shell script portability itself .

[Gilles 'SO- stop being evil']

Testing with dash is the bare minimum to avoid accidental reliance on ksh/bash features, but I'm after more than that: knowing about deficiencies in other shells (e.g. zsh's poor trap handling), and above all in shell utilities (e.g. OpenBSD's find still hasn't implemented -exec +).

[Alex Miller]

That's one of the best resources and was built while writing code in M4 that had to produce shell code that was portable across as many shells as possible.

[Stéphane Chazelas]

Note that the Bourne shell is not a POSIX shell, making your script portable to Bourne shell would mean downgrading the syntax of your script from the standard POSIX sh language to the common denominator between that and the syntax of the Bourne shell. Not worth the opinion unless you want to be portable to ancient systems (or Solaris 10 and earlier and you don't have the choice to use the standard sh which was in /usr/xpg4/bin there).

[tripleee]

Is POSIXLY_CORRECT a GNUism, or more widely supported?

[Kusalananda]

@tripleee It is originally a GNU-ism (see here). I have not seen it outside of GNU systems.