subprocess wildcard usage

Question

import os

import subprocess

proc = subprocess.Popen(['ls','*.bc'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)

out,err = proc.communicate()

print out

This script should print all the files with .bc suffix however it returns an empty list. If I do ls *.bc manually in the command line it works. Doing ['ls','test.bc'] inside the script works as well but for some reason the star symbol doesnt work.. Any ideas ?

Solution 1

You need to supply shell=True to execute the command through a shell interpreter. If you do that however, you can no longer supply a list as the first argument, because the arguments will get quoted then. Instead, specify the raw commandline as you want it to be passed to the shell:

 proc = subprocess.Popen('ls *.bc', shell=True,
                                    stdout=subprocess.PIPE,
                                    stderr=subprocess.PIPE)

Solution 2

Expanding the * glob is part of the shell, but by default subprocess does not send your commands via a shell, so the command (first argument, ls) is executed, then a literal * is used as an argument.

This is a good thing, see the warning block in the "Frequently Used Arguments" section, of the subprocess docs. It mainly discusses security implications, but can also helps avoid silly programming errors (as there are no magic shell characters to worry about)

My main complaint with shell=True is it usually implies there is a better way to go about the problem - with your example, you should use the glob module:

import glob
files = glob.glob("*.bc")
print files # ['file1.bc', 'file2.bc']

This will be quicker (no process startup overhead), more reliable and cross platform (not dependent on the platform having an ls command)

Solution 3

Besides doing shell=True, also make sure that your path is not quoted. Otherwise it will not be expanded by shell.

If your path may have special characters, you will have to escape them manually.

Comments

[jfs]

related: Wildcard not working in subprocess call using shlex

[Cemre Mengü]

Thanks this worked just fine. Some of the examples that I found on the internet had a list as their first argument for some reason

[Niklas B.]

@Cemre: That's usually advisable because you don't want the shell to interpret the arguments. Imagine you pass user input to a command like in 'ls ' + user_supplied_path. The user could just input the path ; shutdown -s and the system would halt! If you use ['ls', user_supplied_path], you prevent this kind of injection.

[Alston]

subprocess does not send your commands via a shell Why? Can you provide any reference? Thanks

[Cyan]

Plus it gives a python object to work with and manipulate. This should be the selected answer. Much more python friendly.

[Alexander C]

In dynamic case it can be very dangerous. Popen("ls " + filename, shell=True), where filename equals blahblah; rm -rf /. Here is my variant

[tripleee]

You probably want to avoid Popen() in favor of subprocess.run() and friends. See also stackoverflow.com/questions/4256107/…

[tripleee]

@Alston The subprocess documentation very clearly documents the meaning of shell=True and how the default is shell=False. See also stackoverflow.com/questions/3172470/…