How to run multiple Python files by running one terminal code?

Question

For my text-mining project (with an Enron data set), I have this Python script:

from glob import glob
import fileinput
with open('/media/output/inbox.txt', 'w') as out:
for line in fileinput.input(glob('*.')):
    if 'Subject:' in line:
        out.write(line)

Inside more than 1000 folders with one-step depth folder structure like this,

-folder
  --folder 2.

by using this terminal command and it works perfectly.

find allen-p fischer-m kitchen-l phanis-s smith-m arnold-j forney-j kuykendall-t pimenov-v solberg-g ,,,,,,,,,,, zufferli-j -maxdepth 1 -exec cp file.py {} \;

By looking at the Python script, I try to extract a particular line from a file and paste it in another file.

Then I realized that I need to run these scripts one by one.

How can I run each script automatically one after another and with all the folder structure?

I am looking for some terminal code like above or a Python script.

Solution 1

You don't need to copy the script to each directory. Just modify the find command to run it instead:

find allen-p ... zufferli-j -maxdepth 1 -type d -execdir python "$PWD"/file.py \;

The -execdir option runs the command inside each directory. We use $PWD to give the full path to the script, so that a changing working directory doesn't affect where the command looks for the script.

However, since you have copied it anyway, you can skip the $PWD:

find allen-p ... zufferli-j -maxdepth 1 -type d -execdir python file.py \;

---

Note

You didn't provide a type check in your find command (like I did with -type d above). Without it, every file in those directories has now been overwritten with file.py. You only want to copy to (and run in) directories, so you should have added a -type d to it.

Solution 2

You could also write a loop that moves into each directory and runs the script before moving back:

for dir in allen-p fischer-m ... solberg-g; do
    cd "$d" && /path/to/python/script.py && ../
done

Note that this will fail if the python script fails in any of the directories.

Comments

[muru]

Which scripts? You only show one.

[Miller]

yes the same script i copy to every folder