Calling multiple bash scripts and running them in parallel, not in sequence
Solution 1
for((i=1;i<100;i++)); do nohup bash script${i}.sh & done
Solution 2
A better way would be to use GNU Parallel. GNU parallel is simple and with it we can control the number of jobs to run in parallel with more control over the jobs.
In the below command, script{1..3}.sh
gets expanded and are sent as arguments to bash
in parallel. Here -j0
indicates that as many jobs should be run as possible. By default parallel
runs one job for one cpu core.
$ parallel -j0 bash :::: <(ls script{1..3}.sh)
And you can also try using
$ parallel -j0 bash ::: script{1..3}.sh
While executing the second method if you get any error message then it means that --tollef
option is set in /etc/parallel/config
and that needs to be deleted and every thing will work fine.
You can read GNU Parallels
man page here for more richer options.
And in case if your are running the jobs from a remote machine, better use screen
so that the session does not gets closed due to network problems. nohup
is not necessary, as recent versions of bash as coming with huponexit
as off
and this will prevent parent shell from sending HUP
signal to its children during its exit. In case if its not unset do it with
$ shopt -u huponexit
Solution 3
We can also use xargs
to run multiple script in parallel.
$ ls script{1..5}.sh|xargs -n 1 -P 0 bash
here each script is passed to bash as argument separately. -P 0
indicates that the number of parallel process can be as much as possible. It is also safer that using bash default job control feature
(&)
.
Solution 4
A single line solution:
$ nohup bash script1.sh & nohup bash script2.sh & nohup bash script3.sh &
Less facetiously, just use a wrapper script:
$ cat script.sh
#!/usr/bin/env bash
script1.sh &
script2.sh &
script3.sh &
$ nohup script.sh &
Or loop over them:
for script in dir/*.sh
do
nohup bash "$script" &
done
Solution 5
If you're looking to save yourself some typing effort
eval "nohup bash "script{1..3}.sh" &"
Or on second thoughts, maybe not
Related videos on Youtube
eikonal
Updated on September 18, 2022Comments
-
eikonal almost 2 years
Suppose that I have three (or more) bash scripts:
script1.sh
,script2.sh
, andscript3.sh
. I would like to call all three of these scripts and run them in parallel. One way to do this is to just execute the following commands:nohup bash script1.sh & nohup bash script2.sh & nohup bash script3.sh &
(In general, the scripts may take several hours or days to finish, so I would like to use
nohup
so that they continue running even if my console closes.)But, is there any way to execute those three commands in parallel with a single call?
I was thinking something like
nohup bash script{1..3}.sh &
but this appears to execute
script1.sh
,script2.sh
, andscript3.sh
in sequence, not in parallel.-
l0b0 over 9 yearsWhat is the use case? Do you have a million scripts to start?
-
eikonal over 9 years@jw013 I mean, something like a single short line command. If I have 100 scripts to start, I would like to be able to type something short (like
nohup bash script{1..100}.sh &
orfor i in {1..100}; do nohup bash script{1..100} &; done
), rather than typingnohup bash script*.sh &
100 different times. -
Hauke Laging over 9 yearsIn case the scripts have useful output: You can start them within
screen
, too (ortmux
), in order to solve the console problem but keep access to the output (and input). -
eikonal over 9 years@l0b0 I have on the order of 100 scripts to start.
-
jw013 over 9 yearsThere is nothing that prevents you from typing all 3 of those commands in the same line.
nohup ... & nohup ... & nohup ... &
. If you mean instead that you want to run all of the scripts without typing each script name individually, a simple loop will do it.
-
-
Mathias Begert over 9 yearsIf you are going to use
bash
as the shellparallel -j0 bash :::: <(ls script{1..3}.sh)
can be reduced toparallel -j0 bash :::: script{1..3}.sh
, no? -
Kannan Mohan over 9 yearsNo its not, when '::::' is used with parallel it means that the argument is a file which contains commands to be executed and not the command itself. Here we are using process substitution to redirect the script names within a files descriptor.
-
Mathias Begert over 9 yearsEr.. in that case why not
parallel -j0 bash ::: script{1..3}.sh
? -
Mathias Begert over 9 yearsIt's
bash ::: script{1..3}.sh
being passed toparallel
, not::: script{1..3}.sh
. So this should first expand toparallel bash ::: script1.sh script2.sh script3.sh
by the shell and thenparallel
invocations ofbash script1.sh
,bash script2.sh
,bash script3.sh
. I tried it -
Kannan Mohan over 9 yearsI think you are confused, Check out the answer I have mentioned
::::
and not:::
, both have different use. Read theparallel
man page to understand more about them. -
Mathias Begert over 9 yearsNo I am not confused, all I am stating is that the OP's problem is better solved with
parallel -j0 bash ::: script{1..3}.sh
- this is better than the::::
approach since it avoids the need for process substitution. Also parsing ls output is ridden with pitfalls -
David Conrad over 9 yearsYou're missing a close parenthesis.
-
Ole Tange over 9 yearsIf you are a
screen
kind of guy, check out--tmux
(version 20140722 or later) which will run each job in its own tmux window. -
Patrick almost 9 years@HaukeLaging What if script names are different ?