How to correctly escape special characters in python subprocess?

Question

Im trying to run this bash command using python subprocess

find /Users/johndoe/sandbox -iname "*.py" | awk -F'/' '{ print $NF}'

output:-

helld.xl.py
parse_maillog.py
replace_pattern.py
split_text_match.py
ssh_bad_login.py

Here is what i have done in python2.7 way, but it gives the output where awk command filter is not working

>>> p1=subprocess.Popen(["find","/Users/johndoe/sandbox","-iname","*.py"],stdout=subprocess.PIPE)

>>> p2=subprocess.Popen(['awk','-F"/"','" {print $NF} "'],stdin=p1.stdout,stdout=subprocess.PIPE)

>>>p2.communicate()
('/Users/johndoe/sandbox/argparse.py\n/Users/johndoe/sandbox/custom_logic_substitute.py\n/Users/johndoe/sandbox/finditer_html_parse.py\n/Users/johndoe/sandbox/finditer_simple.py\n/Users/johndoe/sandbox/group_regex.py\n/Users/johndoe/sandbox/helo.py\n/Users/johndoe/sandbox/newdir/helld.xl.py\n/Users/johndoe/sandbox/parse_maillog.py\n/Users/johndoe/sandbox/replace_pattern.py\n/Users/johndoe/sandbox/split_text_match.py\n/Users/johndoe/sandbox/ssh_bad_login.py\n', None)

I could also get output by using p1 alone here like below,but i cant get the awk working here

list1=[]
result=p1.communicate()[0].split("\n")
for item in res:
    a=item.rstrip('/').split('/')
    list1.append(a[-1])
print list1

Solution

You are incorrectly passing in shell quoting (and extra shell quoting which isn't even required by the shell!) when you're not invoking a shell. Don't do that.

p2=subprocess.Popen(['awk', '-F/', '{print $NF}'], stdin=...

When you have shell=True you need extra quotes around some arguments to protect them from the shell, but there is no shell here, so putting them in is incorrect, and will cause parse errors by Awk.

However, you should almost never need to call Awk from Python, especially for trivial tasks which Python can easily do natively:

list1 = [line.split('/')[-1]
            for line in subprocess.check_output(
                ["find", "/Users/johndoe/sandbox",
                 "-iname", "*.py"]).splitlines()]

In this particular case, note also that GNU find already has a facility to produce this result directly:

list1 = subprocess.check_output(
    ["find", "/Users/johndoe/sandbox",
     "-iname", "*.py", "-printf", "%f\\n"]).splitlines()

Comments

[sherpaurgen]

i could use it but i really want to know how i use awk here which involves special chars like " , { , }, ' , / etc

[tripleee]

You amazingly managed to remove the quotes in the find command line correctly, so it's not clear why you made this error.

[tripleee]

Tangentially see also stackoverflow.com/questions/10067266/…

[tripleee]

/ is not any more "special" than . or a.