Why is it better to use "#!/usr/bin/env NAME" instead of "#!/path/to/NAME" as my shebang?

268,187

Solution 1

Objective Criteria/Requirements:

In determining whether to use an absolute or logical (/usr/bin/env) path to an interpreter in a she-bang, there are (2) key considerations:

a) The interpreter can be found on target system

b) The correct version of interpreter can be found on target system

If we AGREE that "b)" is desirable, we also agree that:

c) It's preferable our scripts fail rather than execute using an incorrect interpreter version and potentially achieve inconsistent results.

If we DON'T AGREE that "b)" matters, then any interpreter found will suffice.

Testing:

Since using a logical path- /usr/bin/env to the interpreter in the she-bang is the most extensible solution allowing the same script to execute successfully on target hosts with different paths to the same interpreter, we'll test it- using Python due to its' popularity- to determine if it meets our criteria.

  1. Does /usr/bin/env live in a predictable, consistent location on POPULAR (not "every") Operating Systems? Yes:
  • RHEL 7.5
  • Ubuntu 18.04
  • Raspbian 10 ("Buster")
  • OSX 10.15.02
  1. Below Python script executed both inside and outside of virtual envelopes (Pipenv used) during tests:
    #!/usr/bin/env pythonX.x
    import sys
    print(sys.version)
    print('Hello, world!')
    
  2. The she-bang in the script was toggled by Python version number desired (all installed on same host):
  • #!/usr/bin/env python2
  • #!/usr/bin/env python2.7
  • #!/usr/bin/env python3
  • #!/usr/bin/env python3.5
  • #!/usr/bin/env python3.6
  • #!/usr/bin/env python3.7
  1. Expected results: that print(sys.version) = env pythonX.x. Each time ./test1.py was executed using a different installed Python version, the correct version specified in the she-bang was printed.

  2. Testing Notes:

  • Tests were exclusively limited to Python
  • Perl: Like Python- MUST live in /usr/bin according to the FHS
  • I've not tested every possible combination on every possible number of Linuxy/Unixy Operating System and version of each Operating System.

Conclusion:

Although it's TRUE that #!/usr/bin/env python will use the first version of Python it matches in the user's Path, we can enforce an express preference by specifying a version number such as #!/usr/bin/env pythonX.x. Indeed, developers don't care which interpreter is found "first", all they care about is that their code is executed using the specified interpreter they know to be compatible with their code to ensure consistent results- wherever that may live in the filesystem...

In terms of portability/flexibility, using a logical- /usr/bin/env - rather than absolute path not only meets requirements a), b) & c) from my testing with different versions of Python, but also has the benefit of fuzzy-logic finding the same version interpreter even if they live at different paths on different Operating Systems. And although MOST distros respect the FHS, not all do.

So where a script will FAIL if binary lives in different absolute path than specified in shebang, the same script using a logical path SUCCEEDS as it keeps going until it finds a match, thereby offering greater reliability & extensibility across platforms.

Solution 2

It isn't necessarily better.

The advantage of #!/usr/bin/env python is that it will use whatever python executable appears first in the user's $PATH.

The disadvantage of #!/usr/bin/env python is that it will use whatever python executable appears first in the user's $PATH.

That means that the script could behave differently depending on who runs it. For one user, it might use the /usr/bin/python that was installed with the OS. For another, it might use an experimental /home/phred/bin/python that doesn't quite work correctly.

And if python is only installed in /usr/local/bin, a user who doesn't have /usr/local/bin in $PATH won't even be able to run the script. (That's probably not too likely on modern systems, but it could easily happen for a more obscure interpreter.)

By specifying #!/usr/bin/python you specify exactly which interpreter will be used to run the script on a particular system.

Another potential problem is that the #!/usr/bin/env trick doesn't let you pass arguments to the intrepreter (other than the name of the script, which is passed implicitly). This usually isn't an issue, but it can be. Many Perl scripts are written with #!/usr/bin/perl -w, but use warnings; is the recommended replacement these days. Csh scripts should use #!/bin/csh -f -- but csh scripts are not recommended in the first place. But there could be other examples.

I have a number of Perl scripts in a personal source control system that I install when I set up an account on a new system. I use an installer script that modifies the #! line of each script as it installs it in my $HOME/bin. (I haven't had to use anything other than #!/usr/bin/perl lately; it goes back to times when Perl often wasn't installed by default.)

A minor point: the #!/usr/bin/env trick is arguably an abuse of the env command, which was originally intended (as the name implies) to invoke a command with an altered environment. Furthermore, some older systems (including SunOS 4, if I recall correctly) didn't have the env command in /usr/bin. Neither of these is likely to be a significant concern. env does work this way, a lot of scripts do use the #!/usr/bin/env trick, and OS providers aren't likely to do anything to break it. It might be an issue if you want your script to run on a really old system, but then you're likely to need to modify it anyway.

Another possible issue, (thanks to Sopalajo de Arrierez for pointing it out in comments) is that cron jobs run with a restricted environment. In particular, $PATH is typically something like /usr/bin:/bin. So if the directory containing the interpreter doesn't happen to be in one of those directories, even if it's in your default $PATH in a user shell, then the /usr/bin/env trick isn't going to work. You can specify the exact path, or you can add a line to your crontab to set $PATH (man 5 crontab for details).

Kevin's comment points out that Python's virtualenv creates a special case, where the environment installs a Python interpreter in a special directory that's inserted at the front of $PATH. For that particular environment (and perhaps others like it), the #!/usr/bin/env python trick (or python3?) is likely to be the best solution. (I haven't used virtualenv myself.)

Solution 3

Because /usr/bin/env can interpret your $PATH, which makes scripts more portable.

#!/usr/local/bin/python

Will only run your script if python is installed in /usr/local/bin.

#!/usr/bin/env python

Will interpret your $PATH, and find python in any directory in your $PATH.

So your script is more portable, and will work without modification on systems where python is installed as /usr/bin/python, or /usr/local/bin/python, or even custom directories (that have been added to $PATH), like /opt/local/bin/python.

Portability is the only reason using env is preferred to hard coded paths.

Solution 4

Specifying the absolute path is more precise on a given system. The downside is that it's too precise. Suppose you realize that the system installation of Perl is too old for your scripts and you want to use your own instead: then you have to edit the scripts and change #!/usr/bin/perl to #!/home/myname/bin/perl. Worse, if you have Perl in /usr/bin on some machines, /usr/local/bin on others, and /home/myname/bin/perl on yet other machines, then you'd have to maintain three separate copies of the scripts and execute the appropriate one on each machine.

#!/usr/bin/env breaks if PATH is bad, but so does almost anything. Attempting to operate with a bad PATH is very rarely useful, and indicates that you know very little about the system the script is running on, so you can't rely on any absolute path anyway.

There are two programs whose location you can rely on on almost every unix variant: /bin/sh and /usr/bin/env. Some obscure and mostly retired Unix variants had /bin/env without having /usr/bin/env, but you're unlikely to encounter them. Modern systems have /usr/bin/env precisely because of its widespread use in shebangs. /usr/bin/env is something you can count on.

Apart from /bin/sh, the only time you should use an absolute path in a shebang is when your script isn't meant to be portable, so you can count on a known location for the interpreter. For example, a bash script that only works on Linux can safely use #!/bin/bash. A script that is only meant to be used in-house can rely on house interpreter location conventions.

#!/usr/bin/env does have downsides. It's more flexible than specifying an absolute path but still requires knowing the interpreter name. Occasionally you might want to run an interpreter that isn't in the $PATH, for example in a location relative to the script. In such cases, you can often make a polyglot script that can be interpreted both by the standard shell and by your desired interpreter. For example, to make a Python 2 script portable both to systems where python is Python 3 and python2 is Python 2, and to systems where python is Python 2 and python2 doesn't exist:

#!/bin/sh
''':'
if type python2 >/dev/null 2>/dev/null; then
  exec python2 "$0" "$@"
else
  exec python "$0" "$@"
fi
'''
# real Python script starts here
def …

Solution 5

Specifically for perl, using #!/usr/bin/env is a bad idea for two reasons.

First, it's not portable. On some obscure platforms env isn't in /usr/bin. Second, as Keith Thompson has noted, it can cause trouble with passing arguments on the shebang line. The maximally portable solution is this:

#!/bin/sh
exec perl -x "$0" "$@"
#!perl

For details on how it works, see 'perldoc perlrun' and what it says about the -x argument.

Share:
268,187

Related videos on Youtube

user1340106
Author by

user1340106

After graduating high school in 1980, and earning my B.S. in Computer Science from Purdue University in 1986, I traveled to Hawai`i to relax. After a decade in the islands building my career and, more importantly, learning of the impractical affects on modern day life resulting from bully-ish behavior of the 18th and 19th century colonialists, I returned to the continent. Having worked for several startups around the country (including a successful one in Silly Valley), I have a yearning for techno-knowledge... of all types.

Updated on September 18, 2022

Comments

  • user1340106
    user1340106 over 1 year

    I notice that some scripts which I have acquired from others have the shebang #!/path/to/NAME while others (using the same tool, NAME) have the shebang #!/usr/bin/env NAME.

    Both seem to work properly. In tutorials (on Python, for example), there seems to be a suggestion that the latter shebang is better. But, I don't quite understand why this is so.

    I realize that, in order to use the latter shebang, NAME must be in the PATH whereas the first shebang does not have this restriction.

    Also, it appears (to me) that the first would be the better shebang, since it specifies precisely where NAME is located. So, in this case, if there are multiple versions of NAME (e.g., /usr/bin/NAME, /usr/local/bin/NAME), the first case specifies which to use.

    My question is why is the first shebang preferred to the second one?

    • jasonwryan
      jasonwryan over 12 years
      See this answer...
    • Pedrom
      Pedrom over 12 years
      @TheGeeko61: In my case I had something broken and some variables wasn't in env. So I suggest to use this shebang to verify if env is correctly loaded.
  • William Pursell
    William Pursell over 12 years
    If /usr/bin/perl is perl 5.8, $HOME/bin/perl is 5.12, and a script requiring 5.12 hardcodes /usr/bin/perl in the shebangs, it can be a major pain to run the script. I've rarely seen having /usr/bin/env perl grab perl from the PATH be a problem, but it is often very helpful. And it is much prettier than the exec hack!
  • Keith Thompson
    Keith Thompson over 12 years
    @WilliamPursell: What happens when someone else who doesn't have your $HOME/bin in their $PATH tries to run the script?
  • William Pursell
    William Pursell over 12 years
    @Keith They cannot run it, but since the script is in $HOME/bin, it's not a problem. If you have privileges to put the script in /usr/bin, then you can upgrade /usr/bin/perl (although that's probably a bad idea.) I've been on boxes where I wanted to install a fairly large package of perl scripts that hard coded /usr/bin/perl in $HOME, did not have root, and had to go through the entire package modifying the shebang. I was most unhappy.
  • Gilles 'SO- stop being evil'
    Gilles 'SO- stop being evil' about 11 years
    IIRC SunOS 4 had /bin symlinked to /usr/bin. I don't have access to a machine to check, but there are several #!/usr/bin/env perl scripts with SunOS 4 support on the web, so I think SunOS 4 does have /usr/bin/env. On the other hand, SunOS 3 probably didn't. Also, IIRC, NeXTSTEP didn't have /bin/env. Sven Mascheck mentions Unicos and SCO OpenServer.
  • Rex See
    Rex See almost 11 years
    It's worth noting that, if you want to use a specific interpreter version, /usr/bin/env is still better. simply because there usually are multiple interpreter versions installed on your machine named perl5, perl5.12, perl5.10, python3.3, python3.32, etc. and if your app has only been tested on that specific version, you can still specify #!/usr/bin/env perl5.12 and be okay even if the user has it installed somewhere unusual. In my experience, 'python' is usually just a symlink to the system standard version (not necessarily the most recent version on the system).
  • Keith Thompson
    Keith Thompson almost 11 years
    @root: For Perl, use v5.12; serves some of that purpose. And #!/usr/bin/env perl5.12 will fail if the system has Perl 5.14 but not 5.12. For Python 2 vs. 3, #!/usr/bin/python2 and #!/usr/bin/python3 are likely to work.
  • Larry Hosken
    Larry Hosken over 10 years
    Custom directories for python executables are particularly common as virtualenv usage increases.
  • Keith Thompson
    Keith Thompson over 10 years
    @GoodPerson: Suppose I write a Perl script to be installed in /usr/local/bin. I happen to know that it works correctly with /usr/bin/perl. I have no idea whether it works with whatever perl executable some random user happens to have in his or her $PATH. Maybe somebody is experimenting with some ancient version of Perl; because I specified #!/usr/bin/perl, my script (which the user doesn't necessarily even know or care is a Perl script) won't stop working.
  • Good Person
    Good Person over 10 years
    You also have no idea if it works with the perl in /usr/bin/perl - it may be a different version. What if the perl version in /usr/bin/perl is updated? What if the user doesn't even have /usr/bin/perl (and instead, more likely, has /usr/local/bin/perl). What if the version of perl in /usr/bin is completely broken and the user has a patched version in ~/opt/bin? Respecting the user's PATH is more correct. If you write code for one very specific never changing system, I could see you relying on the quirks of the system. Otherwise, it is best to be portable
  • Keith Thompson
    Keith Thompson over 10 years
    If it doesn't work with /usr/bin/perl, I'll find out very quickly, and it's the system owner/administrator's responsibility to keep it up to date. If you want to run my script with your own perl, feel free to grab and modify a copy or invoke it via perl foo. (And you might consider the possibility that the 55 people who upvoted this answer also know a thing or two. It's certainly possible that you're right and they're all wrong, but that's not the way I'd bet.)
  • AmokHuginnsson
    AmokHuginnsson almost 9 years
    Exactly the anser I was looking for: how to write porable "shebang" for perl script that will allow additional aruments passed to perl (last line in your example accepts additional aruments).
  • jlliagre
    jlliagre over 8 years
    Both of these potential issues have already been mentioned in existing answers. In any case, env is much more likely to be in /usr/bin/env than bash or python being in any specific location.
  • Alessio
    Alessio over 8 years
    Nobody else here has mentioned the ARGV[0] problem. And nobody has mentioned the /path/to/env issue in a form that directly addresses one of the arguments for using it (i.e. that bash or perl might be in an unexpected location).
  • jlliagre
    jlliagre over 8 years
    Sorry, you are right about the argv[0] issue, I was confusing with another thread. Your first point is still questionable. Using env in the first place assumes the command to launch is in the user's PATH and is a portable way to publish a script giving it the best opportunity to be executable whatever the target system. If that target command (e.g python, perl) is not in the PATH, there is no portable way to specify it in the script shebang anyway.
  • Alessio
    Alessio over 8 years
    The interpreter path issue is easily fixed by a sysadmin with symlinks, or by the user editing their script. It's certainly not a significant enough problem to encourage people to put up with the all of the other issues caused by using env on the shebang line that are mentioned here and on other questions. That's promoting a bad solution in the same kind of way that encouraging csh scripting is promoting a bad solution: it kind of works but there are much better alternatives.
  • jlliagre
    jlliagre over 8 years
    Not everybody is a sysadmin on their machine. The env solution is aimed at helping non operating system specialists to copy/paste or download scripts that have a good chance to work as is. It is also one of the methods that allow more advanced users to experiment with multiple unchanged scripts to be executed by custom versions of the target interpreter by only changing their PATH / PATH order (sorry about that too long sentence...)
  • Alessio
    Alessio over 8 years
    non-sysadmins can ask their sysadmin to do it. or they can simply edit the script and change the #! line.
  • jlliagre
    jlliagre over 8 years
    That is precisely what env is helping to avoid: depending on a sysadmin or OS specific technical knowledge unrelated to python or whatever.
  • Kevin
    Kevin over 8 years
    This answer is not congruent with how Python in particular is commonly used. The Python executable you want to use is rarely /usr/bin/python. Typically, the one you want lives in a Virtualenv, and is at the front of the user's $PATH. In the rare situations where that is not the case, /usr/bin will usually be on the user's $PATH anyway.
  • Keith Thompson
    Keith Thompson over 8 years
    @Kevin: Hmm. I'm not much of a Python programmer myself, but I find your statement surprising. I would have thought that most users of Python scripts (not necessarily Python programmers) don't even use Virtualenv. On the other hand, I suppose the #!/usr/bin/env hack would still work for such users -- but it wouldn't have much advantage over #!/usr/bin/python.
  • Kevin
    Kevin over 8 years
    @KeithThompson: Most uses of Python are in servers. Those uses require isolation and must not interfere with each other. Because of the highly dynamic nature of Python, this isolation becomes very important and cannot be relegated to the OS level. You may have multiple separate web applications running on Python, and they may have totally disjoint support situations (e.g. one only runs on 2.x and one only runs on 3.x). The Windows version of Python explicitly supports the /usr/bin/env hack.
  • Keith Thompson
    Keith Thompson over 8 years
    @Kevin: Just one small point: I've never seen /usr/bin/python be a Python 3 executable; that's generally /usr/bin/python3. (Unless some newer distributions have changed that -- which would break a lot of scripts.)
  • Kevin
    Kevin over 8 years
    @KeithThompson: Arch did and it broke everything.
  • Nathan Basanese
    Nathan Basanese over 8 years
    // , Excellent idea. The whole point of multiple interpreters is NOT to break the code, or to have code depend upon that specific interpreter.
  • Krazy Glew
    Krazy Glew almost 8 years
    "In all other cases use #!/usr/bin/env" is too strong. // As @KeithThompson and others point out, #!/usr/bin/env means the script can behave differently depending on who/how run. Sometimes this different behavior can be a security bug. // For example, NEVER use "#!/usr/bin/env python" for a setuid or setgid script, because the user invoking the script can place malware in a file called "python" in PATH. Whether setuid/gid scripts are ever a good idea is a different issue - but certainly, no setuid/gid executable should ever trust the user provided environment.
  • x-yuri
    x-yuri almost 8 years
    @Kevin: What exactly it broke? Considering it still points to python3, it all went well.
  • x-yuri
    x-yuri almost 8 years
    I've been using it about 5 years. Didn't have any python issues yet.
  • Anthony
    Anthony about 7 years
    One point of interest that I've run into is when pointing an interpreter at a script versus executing the script directly. Example: in my environment, my path loads /Applications/XAMPP/bin/php, so if I run php someTestScript.php, it will be run by the XAMPP php everytime, but if I use ./someTestScript.php and the shebang is set to #!/usr/bin/php, the script is executed by /usr/bin/php. So there can be some confusion (from a user standpoint) regarding why a script has different output based on php someTestScript.php vs ./someTestScript.php
  • Keith Thompson
    Keith Thompson about 7 years
    @Anthony: True -- but not particularly relevant in my experience. The whole point of the #! line is that you can invoke the script as an ordinary command; you don't have to care whether it's run by php, sh, perl, or it's a binary executable. It's unlikely that I'd type php someTestScript.php. But of course other people's habits might differ.
  • Anthony
    Anthony about 7 years
    using php someTestScript.php is handy when not wanting to make the file explicitly executable, and when wanting to run it without a shebang (like if you just wanted to confirm what a script outputs or if it throws errors, etc). But many times I've seen scripts that have a shebang line to make them executable (basically so it can get some first-class cred), but documentation will still be mixed on how to execute (some docs using php someTestScript others using ./someTestScript)....
  • Anthony
    Anthony about 7 years
    ... You're right that it shouldn't matter (and obviously, documentation should be better written, etc), but the part of this answer "the script could behave differently depending on who runs it" reminded me of this similar scenario, where a script might run differently than expected (or understood) due to a different interpreter being used.
  • Arlene Mariano
    Arlene Mariano almost 7 years
    What about using the env way on CRON tabs?
  • Keith Thompson
    Keith Thompson almost 7 years
    @SopalajodeArrierez: Um, what about it? If you mean, for example, using /usr/bin/env python some_script as a command in a crontab, that's no different than python some_script; both the env command and the cron daemon use the same $PATH.
  • Arlene Mariano
    Arlene Mariano almost 7 years
    Well, I was trying * * * * * /usr/bin/env bash /path/to/MyScript.sh on my CronTab, but, even when any .sh script that starts by #!/usr/bin/env bash seems to work fine, the referred MyScript.sh is not executed at all. Possible explanation: my Linux is a Fun_Plug (BusyBox modification) one running on a NAS. The only way to make it work has been * * * * * /ffp/bin/bash /path/to/MyScript.sh.
  • Keith Thompson
    Keith Thompson almost 7 years
    @SopalajodeArrierez: The /usr/bin/env trick depends on the command being in your $PATH. Apparently your system doesn't have bash in one of the directories in cron's restricted $PATH. Dropping the /usr/bin/env would have no effect. But if you edit your script with #!/ffp/bin/bash, you can execute it directly: * * * * * /path/to/MyScript.sh.
  • Keith Thompson
    Keith Thompson almost 7 years
    @SopalajodeArrierez: I've updated my answer to address your point (see the last paragraph). Thanks!
  • ckujau
    ckujau almost 7 years
    +1 for mentioning that env doesn't always reside in /usr/bin (although it's pretty exotic).
  • Ezra
    Ezra over 6 years
    I wish I could upvote this a thousand times, because it's actually a great answer either way -- if someone uses /usr/bin/env and I need to get rid of it locally, or if they didn't use it and I need to add it. Both cases can occur, and therefore the scripts provided in this answer are a potentially useful tool to have in the toolbox.
  • kristianp
    kristianp about 6 years
    What about #! python, why isn't that used?
  • Tim Kennedy
    Tim Kennedy about 6 years
    #! python isn't used because you'd have to be in the same directory as the python binary, since the bareword python is interpreted as the complete path to the file. If you don't have a python binary in the current directory, you'll get an error like bash: ./script.py: python: bad interpreter: No such file or directory. It's the same as if you used #! /not/a/real/path/python
  • MayeulC
    MayeulC over 5 years
    @KrazyGlew, you can't setuid a script on Linux. You do it trough an executable. And when writing this executable, it is good practice, and widely done, to clear the environment variables. Some environment variable are also purposefully ignored.
  • xdhmoore
    xdhmoore over 4 years
    env is no more guaranteed to be in /usr/bin/env than bash is guaranteed to be in /bin/bash or python in /usr/bin/python - This seemed right at first but then I thought, what is the likelihood of me or someone else putting env in a weird place vs putting python in a weird place? I'm much more likely to put python in a wacky place, so env does seem more predictable to me.
  • user1340106
    user1340106 about 4 years
    Excellent analysis. We appreciate it.
  • Keith Thompson
    Keith Thompson about 4 years
    The question wasn't specific to Python. I don't think you can safely assume that other interpreters will be installed as, for example interpX and interpX.x. For example, the system I'm using at the moment has perl, perl5.26.3, and perl5.30.1. Another has perl and perl5.26.1. Neither has a perl5 command. Also, you didn't mention where the pythonX and pythonX.x interpreters on any of the systems you tested were installed. If they were all in /usr/bin then your experiment doesn't demonstrate any advantage of #!/usr/bin/env pythonX.x over #!/usr/bin/pythonX.x.
  • F1Linux
    F1Linux about 4 years
    @KeithThompson Chose Python for example due to it's huge popularity. The other answers were general and I wanted to take a more practical approach to answering the question. As specifically relates to Perl- I note in my updated answer (responding to your valid concerns) in "Testing">"Notes"- this too must live in /usr/bin according to the FHS. Although I though it was implied in my answer, added a pp in Conclusions that where both options would yield same result the logical path has added benefit of offering greater reliability where binaries might live in diff paths on diff systems
  • vonbrand
    vonbrand about 4 years
    @AmokHuginnsson it is a bletcherous kludge.
  • vonbrand
    vonbrand about 4 years
    @xdhmoore, /usr/bin/env is dependable, where the heck python lives is anybody's guess (most Linux systems have /usr/bin/python, on non-Linux systems it is usually an "unsupported addon" buried somewhere funky).
  • vonbrand
    vonbrand about 4 years
    To paraphrase, "not all the world's a Linux". And not all Linux systems abide by hier(7) either. Most non-Linux systems just don't have bash, or it is a "unsupported, unofficial addon" buried somewhere obscure.
  • DrHyde
    DrHyde about 4 years
    It is, but it's a bletcherous kludge that works. It's a blethcherous kludge that exists because less kludgey solutions don't work.
  • jouell
    jouell about 4 years
    the first 3 sentences say it all!
  • Jonathan Klabunde Tomer
    Jonathan Klabunde Tomer about 4 years
    @TimKennedy this is demonstrably false on my system: pastebin.com/tjV3yARh
  • Jonathan Klabunde Tomer
    Jonathan Klabunde Tomer almost 4 years
    @TimKennedy aha, from man zsh-misc: "If execution fails because the file is not in executable format, and the file is not a directory, it is assumed to be a shell script. /bin/sh is spawned to execute it. If the program is a file beginning with `#!', the remainder of the first line specifies an interpreter for the program. The shell will execute the specified interpreter on operating systems that do not handle this executable format in the kernel." So zsh has implemented its own shebang-line parsing in case it runs on a non-POSIX system, and its implementation differs from Linux kernel's.
  • LLlAMnYP
    LLlAMnYP almost 4 years
    What if sh is not in /bin? True story for google distroless containers.
  • DrHyde
    DrHyde almost 4 years
    Yeah, it also won't work if perl isn't in the path, or on an Amiga, or ... If you're using something weird like that then you're going to have to do some work yourself.
  • DrHyde
    DrHyde almost 4 years
    I wish I could upvote this a thousand times for demonstrating use of ed(1) in a pipeline!
  • Teemu Leisti
    Teemu Leisti over 3 years
    UPVOTED for the formatting alone, never mind the very useful answer.
  • F1Linux
    F1Linux over 3 years
    @TeemuLeisti I try to save other technologists the effort of solving the same problems I bump into. So it's nice to hear somebody found my solution useful, especially when it's involved a fair amount of testing & documentation such as this one has. Most obliged for your kind words!
  • AleXoundOS
    AleXoundOS almost 3 years
    Splitting arguments and passing them to the interpreter works, for example: #! /usr/bin/env -S bash -c 'nix-shell --pure $0'. Also shebang is not meant to pickup the exact versions of software. Other tools manage this nowadays, for example Nix, or other sandboxing/containerization tooling.
  • Keith Thompson
    Keith Thompson almost 3 years
    @AleXoundOS In my experience the handling of arguments on a #! line can vary across systems. On my system, the remainder of the line after the command seems to be treated as a single argument. And those other tools don't do much good if they're not available on the target system.
  • Philip Couling
    Philip Couling almost 3 years
    @KeithThompson it's a long time later... but I'd definitely second Kevin's comments on python here. If you expect the code to be packaged up in a .rpm or .deb and installed to /usr/bin then then #!/usr/bin/python3 is the way to go. That's because it's expected to behave as a system utility and use the system interpreter. But that's pretty much the only time you use the system interpreter. In (nearly) every other context including PyPi you use a virtual environment and you need #!/usr/bin/env python or the scripts won't install or run correctly.
  • Keith Thompson
    Keith Thompson almost 3 years
    @PhilipCouling Thanks, I've added the information from Kevin's comment to my answer.
  • Mikko Rantalainen
    Mikko Rantalainen about 2 years
    @JonathanKlabundeTomer Very interesting. It seems that zsh has special code branch that catches the error about exec() failing and it will try to workaround that by doing extra magic. I wish the Linux native exec() handling were changed to allow shabang to start with non-slash character to execute a binary from PATH because trusting zsh specific error handling to avoid writing /usr/bin/env or absolute path doesn't seem worth the effort. I guess it's too late to fix this, though, because non-absolute path already means relative to script file, not relative to PATH.
  • Mikko Rantalainen
    Mikko Rantalainen about 2 years
    I fully agree with this answer and I just want to mention that the real problem is people writing /usr/bin/python instead of /usr/bin/python2 or /usr/bin/python3 because those are two different programming languages! The /usr/bin/python should have been re-defined to always point to python2 compatible version and we wouldn't even have this whole mess. The same with ruby – if you make incompatible variant, change the name of the interpreter binary, don't mess with the PATH.
  • Mikko Rantalainen
    Mikko Rantalainen about 2 years
    Or maybe you could use different interpreter name if you change the language? If you put env ruby at the front of your script and you actually require some specific version, you're doing it backwards. The env ruby should mean any version of ruby is okay. Both python and ruby have this same problem where the language syntax or features is different but different people pretend to use the same language and use identical shebang.
  • Alessio
    Alessio about 2 years
    @MikkoRantalainen unfortunately, it's more complicated than that for python. python manages to have incompatibilities even within the same major version....and complicated even further by python expecting separate lib directories for each minor version too. It's like they go out of their way to ensure breakage on every release.
  • Mikko Rantalainen
    Mikko Rantalainen about 2 years
    @cas Oh... I wasn't aware that python is incompatible even between minor versions, too. That's one more reason not to use python for anything important.