How can I run multiple Bash scripts simultaneously in a terminal window?

Question

I need to run several Bash scripts (the same script with different variables to be precise) at the same time. To keep the number of tabs under control, I wish to group them in a single terminal tab.

The scripts regularly output, which I check for any problem.

If I send them to the background as

./script.sh 1 &
./script.sh 2 &
./script.sh 3 &
./script.sh 4

I will lose control over them. For example, I terminate the script by Ctrl+C. With the above code, I should find the pid for each process to kill them.

Note that the above code is the content of my main script (say ./all_scripts.sh) rather than commands to be typed in the terminal.

Is there a way to run the script in the same terminal while treating them as a single outputting script?

Solution 1

After testing different methods and programs, I found that the pragmatic solution is GNU Parallel. I post this answer as it may help others.

GNU Parallel has not been built for this task, but perfectly serves the purpose.

If running the scripts as

parallel -u ::: './script.sh 1' './script.sh 2' #(and so forth)

All scripts will be run in parallel.

The -u (--ungroup) flag sends the script outputs into stdout while executing the scripts.

Ctrl+C kills the parallel job, and subsequently all running scripts.

Solution 2

You don't lose control over them. You're sending them (other than the last one) to the background.

If you run the command chain you specify, the invocation ./script.sh 4 will be in the foreground, and the other scripts will be running in the background. Input will be sent to the foreground script. To suspend the foreground script, press CtrlZ. To send the suspended script to the background to continue running, use the bg command.

To see the scripts (or more correctly, jobs) you have and the states they're in, use jobs.

To bring a specific job to the fore, use fg and its number (as reported by the aforementioned jobs) with a % prefix, e. g. fg %2. To terminate a specific job, you can either bring it to the foreground with fg and terminate it sanely, or you can kill it, e. g. kill -TERM %2.

Solution 3

You can redirect the output to a file (one per process or a combined one). Having one file for all is not a problem if the writes are small (512 byte is safe, probably even 4K). Small writes are atomic:

./script.sh 1 >output.1 2>&1 &
./script.sh 2 >output.2 2>&1 &
./script.sh 3 >output.3 2>&1 &
./script.sh 4 >output.4 2>&1 &
./script.sh 5 >output.5 2>&1 &

while wait % >/dev/null 2>&1; do : ; done

If you are doing that interactively then the following may help: You can easily address the jobs by prepending a (otherwise useless) environment variable definition (like BackGround Task ID):

bgtid=foo ./script.sh 1 &
[...]
kill %bgtid=foo

The command prefix just must be unambigeous (so no bgtid=foo and bgtid=foobar).

Solution 4

You can use tmux for this.

It is a terminal multiplexer meaning that it splits one tab into multiple windows.

• Start it with the command tmux.
• Use Contr+B followed by " or % in order to split a pane into two panes.
• Start processes in the foreground.
• Switch between the processes uding Contr+B followed by arrow keys.

Now you have normal control over multiple processes in one tab of your terminal.

If you want to focus (or unfocus) a specific pane, use Contr+B followed by Z.

If you need to scroll use Contr+B followed by Q and scroll using arrow keys(or activate mouse mode).

You can find a cheatsheet here.

It is also possible to automate that process.

Solution 5

echo -e "1\n2\n3\n4\n" | xargs -n1 -P4 ./script.sh

Is another way of doing this. xargs is available on basically any system, and control-c should kill all the jobs.

It also (unlike invoking your scripts with &, allows you to run this from a higher-level bash script)

The -n1 option tells xargs to run each argument in it's own process, and the -P4 option tells it to run 4 processes.

View more solutions

Comments

[jordanm]

You don't need to find the pid, you can use built-in job control eg kill %1

[Googlebot]

Your solution is subtle and solves my problem, but I have a tiny problem to implement it. This works when typing the above commands in the terminal. However, I need to put all commands in a single script named all_scripts.sh. When running this script, the output of all scripts come to the foreground, but job return [1]+ Stopped ./all_scripts.sh, and bg sends the main script to the background instead of the last one in the list.

[Pankaj Goyal]

Job control is intended for use in an interactive context, and is disabled when there is no such context (i. e. a script being executed). You may need to get creative with the use of screen, tmux, ansible playbooks, or other options.

[Pankaj Goyal]

That said, if all you need to do is run the scripts in parallel and wait for them to complete, you can do something like ./script.sh 1 & ./script.sh 2 & ./script.sh 3 & ./script.sh 4 & wait in your script.

[Jerb]

Actually I think parallel was built for exactly this use case. You say it's the same script you're running but with different parameters, right? That's what parallel does. The way you listed it here is fine, but you may also want to consider something like parallel -u ./script.sh ::: 1 2 #(and so forth). Also see the --line-buffer option.