python- run script on multiple files

Question

I have a python script which takes the filename as a command argument and processes that file. However, i have thousands of files I need to process, and I would like to run the script on every file without having to add the filename as the argument each time.

The script works well when run on an individual file like this:

myscript.py /my/folder/of/stuff/text1.txt

I have this code to do them all at once, but it doesn't work

for fname in glob.iglob(os.path.join('folder/location')):
    proc = subprocess.Popen([sys.executable, 'script/location.py', fname])
    proc.wait()

Whenever I run the above code, it doesn't throw an error, but doesn't give me the intended output. I think the problem lies with the fact that the script is expecting the path to a .txt file as an argument, and the code is only giving it the folder that the file is sitting in (or at least not a working absolute reference).

How to correct this problem?

Solution

If the files are in the same folder and if the script supports it, you could use that syntax :

myscript.py /my/folder/of/stuff/*.txt

The wild card will be replaced by the corresponding files.

If the script doesn't support it, isolate the process like in this quick example :

import sys

def printFileName(filename):
  print filename

def main():
  args = sys.argv[1:]
  for filename in args:
    printFileName(filename)

if __name__ == '__main__':
  main()

Then from the console, you can start it like that :

python MyScript.py /home/andy/tmp/1/*.txt /home/andy/tmp/2/*.html

This will print the pathes of all the files in both folders.

Hope this can be of some help.