Use & (ampersand) in single line bash loop
Solution 1
Drop the ;
after &
. This is a syntactic requirement
for((i=114;i<=255;i+=1)); do echo $i > numbers.txt;python DoMyScript.py & done
Solution 2
Given Stephane's comment on 1_CR's answer, you probably want:
for i in {114..255}; do { echo $i > numbers.txt && python DoMyScript.py; } & done
Solution 3
Lose the ;
:
for((i=114;i<=255;i+=1)); do echo $i > numbers.txt;python DoMyScript.py & done
Solution 4
As stated by the other answers:
- The problem is the
;
after the&
. - Deleting the
;
after the&
will result in a command that will run without shell syntax errors. However, it is unlikely to function correctly, because it creates a race condition.
The OP stated that he resolved the race condition by adding a 3 second delay.
Nobody mentioned:
- A 3 second delay might work today. Tomorrow, the computer might be more sluggish, and it could fail again. Next year, the script might be modified to be more time-consuming, and it could fail again. The probability of failure goes down if you increase the delay by an order of magnitude, by which I mean 30 seconds. Of course this means that the script processes will (probably / presumably) be running essentially sequentially, and not concurrently (in parallel), thus defeating the purpose of making the processes asynchronous.
- If running multiple instances of the script is a requirement,
it’s probably best to modify the script to accept the number parameter
on the command line:
and do away with the file.for ((i=114;i<=255;i++)); do python DoMyScript.py --number="$i" & done
- If changing the script is not an option, this solution should work:
This gives each script process a separatefor i in {114..125}; do ( subdir="dir.$i" && mkdir "$subdir" && cd "$subdir" && echo "$i" > numbers.txt && python ../DoMyScript.py; cd .. && rm -r "$subdir" ) & done
numbers.txt
file, by giving it a separate directory to run in. The directory is created and deleted, and the script is run, in a subshell.-
If
DoMyScript.py
accesses any files other thannumbers.txt
by relative pathnames, the above command will need to be adjusted to accommodate that. -
The above was inspired by glenn jackman’s answer (although I believe that mine will work and his won’t).
-
If command
B
depends on the success of commandA
, thenA && B
is better defensive programming thanA; B
. But it’s probably not the ideal way to handle this situation. Ifmkdir dir.114
fails, then the next 111 attempts are likely to fail also, and you’ll get 112 error messages. It would be better to abort the loop if a fatal error occurs.The fact that the action is happening in an asynchronous subshell makes this somewhat tricky.
for i in {114..125}; do { subdir="dir.$i" && mkdir "$subdir" && echo "$i" > "$subdir"/numbers.txt; } || break; ( cd "$subdir" && python ../DoMyScript.py; cd .. && rm -r "$subdir" ) & done
will cause the loop to abort if a
mkdir
or anecho value > file
command fails. -
It might be better to work in a directory that is ‘‘guaranteed’’ to be writable, like
/tmp
. However, this increases the risk that your command will collide (interfere) with some other process.- You can mitigate this by adding
$$
to the directory name; e.g.,dir.$$.$i
or evendir.$BASHPID
. - This would not eliminate the risk that the
mkdir
or the file creation might fail because the filesystem is full.
- You can mitigate this by adding
-
Note that the above code will keep on going and remove the temporary directory even if the script fails. You might want to do something else in that situation.
- Oops.
If, for some reason,
the
mkdir "$subdir"
succeeds but thecd "$subdir"
fails, this will go ahead and docd .. && rm -r "$subdir"
. If there is a directory with a name likedir.114
in your parent directory (i.e., parallel to your current directory), it will be removed. I believe that you can (at least somewhat) fix this by changing the last line to
or by using absolute paths; e.g.,( cd "$subdir" && { python ../DoMyScript.py; cd ..; } && rm -r "$subdir" ) & done
subdir="$PWD/dir.$i"
.
- Oops.
If, for some reason,
the
-
Related videos on Youtube
ljs.dev
Updated on September 18, 2022Comments
-
ljs.dev over 1 year
I have been using this command successfully, which changes a variable in a config file and then executes a Python script within a loop:
for((i=114;i<=255;i+=1)); do echo $i > numbers.txt; python DoMyScript.py; done
As each
DoMyScript.py
instance takes about 30 seconds to run before terminating, I'd like to relegate them to the background while the next one can be spawned.I have tried what I am familiar with, by adding in an ampersand as below:
for((i=114;i<=255;i+=1)); do echo $i > numbers.txt; python DoMyScript.py &; done
However, this results in the below error:
-bash: syntax error near unexpected token `;'
-
Martin von Wittich over 10 yearsDamn it, 43 seconds too slow :)
-
Mathias Begert over 10 years@MartinvonWittich, a +1 to your answer by way of commiseration ;-)
-
Stéphane Chazelas over 10 yearsWhile that answers the question, it's probably not what the OP wants as by the time all the python instances have started and initialises they'll all see a
numbers.txt
containing255
. -
user over 10 yearsAgreed with @StephaneChazelas, but am not sure what we can really do about it with the information given in the question. A proper solution to that probably needs more context.
-
ljs.dev over 10 yearsThat worked great. OP likes the outcome, the file is changed and then read each time, so each iteration of the script gets a different variable. I did need to add in a 3 second delay just to avoid some race-condition/oddities.
-
Angel Todorov over 10 yearsHere's the bash manual reference: gnu.org/software/bash/manual/bashref.html#Lists
-
ljs.dev over 10 yearsThanks Glenn. I tried a few variations of your code which I like due to the increased readability, but perhaps due to the extra sleep I needed, it was not working out for me. It would also repeat the last number in the
for
range... Here's what I am happy with at the moment:for((i=1;i<100;i+=1)); do echo $i > numbers.txt;sleep 3; python DoMyScript.py & done
The main question point of getting the process to execute in the background is solved. -
ljs.dev over 10 yearsThanks @glennjackman, I would not have thought to search under
lists
, based on my previous usage of&
limited to just appending it to commands to shush them. Time to bash my head against bash properly! -
ljs.dev over 10 years*of course, I had
<=
in the original question, so the range you supplied was totally suitable based on the info I supplied. Again, thanks for the different approach -
G-Man Says 'Reinstate Monica' almost 4 yearsChanging the
;
afterecho $i > numbers.txt
to&&
is an improvement. OTOH, if you can’t write a three-digit number to a file, you should probably abort the whole operation, and if you can write to the file on some iterations but not others, you have a problem that needs to be investigated and resolved. … (Cont’d) -
G-Man Says 'Reinstate Monica' almost 4 years(Cont’d) … But, aside from that, I don’t see how this offers any real improvement. You still have 112
python DoMyScript.py
processes being spawned milliseconds apart. UnlessDoMyScript.py
starts very quickly, the shell loop will have gone on and written the next value tonumbers.txt
beforeDoMyScript.py
has read the value it’s supposed to read.