Subprocess module fails to run command

Question

I'm trying to execute Google's cpplint.py on a group of my files and collect the results to one log file. However, I have not managed to beat the subprocess module. My current code is here:

import os, subprocess

rootdir = "C:/users/me/Documents/dev/"
srcdir = "project/src/"

with open(rootdir+srcdir+"log.txt", mode='w', encoding='utf-8') as logfile:
    for subdir, dirs, files in os.walk(rootdir+srcdir):
        for file in files:
            if file.endswith(".h") or file.endswith(".cpp"):
                filewithpath=os.path.join(subdir, file)
                cmd=['c:/Python27/python.exe','C:/users/me/Documents/dev/cpplint.py','--filter=-whitespace,-legal,-build/include,-build/header_guard/', filewithpath]               
                output = subprocess.check_output(cmd)
                logfile.write(output.decode('ascii'))

Trying to run the above code throws an error:

  File "C:\Python32\lib\site.py", line 159
    file=sys.stderr)
        ^ SyntaxError: invalid syntax Traceback (most recent call last):   File "C:\Users\me\Documents\dev\project\src\verifier.py", line 19, in <module>
    output = subprocess.check_output(cmd)   File "C:\Python32\lib\subprocess.py", line 511, in check_output
    raise CalledProcessError(retcode, cmd, output=output) subprocess.CalledProcessError: Command '['c:/Python27/python.exe', 'C:/users/me/Documents/dev/cpplint.py', '--filter=-whitespace,-legal,-build/include,-build/header_guard/', 'C:/users/me/Documents/dev/project/src/aboutdialog.cpp']' returned non-zero exit status 1

If I substitute the cmd with something simpler like:

cmd=['C:/WinAVR-20100110/bin/avr-gcc.exe','--version']

Then the script works as expected.

I have also tried to use a single command string instead of a list of strings as cmd, but the result is the same. When debugging the code, I copied the list-of-strings-turned-into-the-command-line-command from the debugger and ran it in the Windows command line, and the command ran as expected.

The Python interpreter running my script is Python 3.2. Any tips are greatly appreciated.

Solution 1

Looks like cpplint.py is simply exiting with a non-zero return code - which it might do, for instance, if it finds errors or "lint" in the source files it is checking.

See the documentation for subprocess.check_output. Note that if the command executed returns a non-zero exit code then a subprocess.CalledProcessError is raised.

You could work around it by watching for CalledProcessError, e.g.

try:
    output = subprocess.check_output(cmd)
except subprocess.CalledProcessError as e:
    # ack!  cpplint.py failed... report an error to the user?

EDIT:

The SyntaxError seems to be the key here, and is probably caused by C:\Python32\lib being in your PYTHONPATH (either explicitly, or, this could happen if it is your current working directory).

The Python interpreter (since about 1.5.2-ish) automatically runs import site when started. So, when this is the case, and your script goes to execute:

c:/Python27/python.exe C:/users/me/Documents/dev/cpplint.py ...

then the Python 2.7 interpreter will find C:\Python32\lib\site.py first, and try to load that, instead of the one (presumably) at C:\Python27\lib\site.py. The issue is that Python 3's site.py contains syntax incompatible with Python 2, so the process launched by subprocess.check_output is failing with a SyntaxError before it even gets a chance to run cpplint, which propagates the CalledProcessError.

Solution? Make sure Python2 gets a bonafide Python2 "PYTHONPATH", and likewise for Python3! In other words, make sure C:\Python32\lib is not in the PYTHONPATH search path when running the Python2 interpreter.

One way to do this in your case is to set an explicit environment when launching the process, e.g.:

python2_env = {"PYTHONPATH": "path/to/python2/stuff:..."}
output = subprocess.check_output(cmd, env=python2_env)

Solution 2

I would request you to run this first

pipe = subprocess.Popen([cmd, options],stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
stdout, stderr = pipe.communicate()

You will get to know what exactly is the error behind it, since CalledProcessError is raised only if the exit code was non-zero.

Solution 3

I did it by replacing the def main() with the following (I editet the errorfunction too to get a proper csv-file):

errorlog = sys.stderr
sys.stderr = open("errorlog.csv","w")
sys.stderr.write("File;Line;Message;Category;Confidence\n")
for filename in filenames:
  ProcessFile(filename, _cpplint_state.verbose_level)
_cpplint_state.PrintErrorCounts()
sys.exit(_cpplint_state.error_count > 0)
sys.stdout = errorlog
sys.stderr.close()