Putting subshell in background vs putting command in background

Question

I have two bash scripts that try to check hosts that are up:

Script 1:

#!/bin/bash 
for ip in {1..254}; do
    ping -c 1 192.168.1.$ip | grep "bytes from" | cut -d" " -f 4 | cut -d ":" -f 1 &
done

Script 2:

#!/bin/bash 
for ip in {1..254}; do
    host=192.168.1.$ip
    (ping -c 1 $host > /dev/null 
    if [ "$?" = 0 ]
    then 
        echo $host
    fi) &
done

As I am checking a large range, I would like to process each ping command in parallel. However, my second script seems to not retry failed fork attempts due to resource limits. This results in the second script having inconsistent results while my first script gives constant results despite both failing to fork at times. Can someone explain this to me? Also is there anyway to retry failed forks?

Solution 1

There is already an answer which gives an improved code snippet to the task the original poster questions was related to, while it might not yet have more directly responded to the question.

The question is about differences of

• A) Backgrounding a "command" directly, vs
• B) Putting a subshell into the background (i.e with a similar task)

Lets check about those differences running 2 tests

# A) Backgrounding a command directly
sleep 2 & ps

outputs

[1] 4228
  PID TTY          TIME CMD
 4216 pts/8    00:00:00 sh
 4228 pts/8    00:00:00 sleep

while

# A) backgrounding a subhell (with similar tas)
( sleep 2; ) & ps

outputs something like:

[1] 3252
  PID TTY          TIME CMD
 3216 pts/8    00:00:00 sh
 3252 pts/8    00:00:00 sh
 3253 pts/8    00:00:00 ps
 3254 pts/8    00:00:00 sleep

** Test results:**

In this test (which run only a sleep 2) the subshell version indeed differs, as it would use 2 child processes (i.e. two fork()/exec operations and PID) and hence more than the direct backgrounding of the command.

In the script 1 of the question however the command was not a single sleep 2s but instead it was a pipe of 4 commands, which if we test in an additional case

• C) Backgrounding a pipe with 4 commands

# C) Backgrounding a pipe with 4 commands
sleep 2s | sleep 2s | sleep 2s | sleep 2s & ps

yields this

[2] 3265
  PID TTY          TIME CMD
 3216 pts/8    00:00:00 bash
 3262 pts/8    00:00:00 sleep
 3263 pts/8    00:00:00 sleep
 3264 pts/8    00:00:00 sleep
 3265 pts/8    00:00:00 sleep
 3266 pts/8    00:00:00 ps

and shows that indeed the script 1 would be a much higher strain in terms of PIDs and fork()s.

As a rough estimate the script one would have used about 254 * 4 ~= 1000 PIDs and hence even more than the script 2 with 254 * 2 ~= 500 PIDs. Any problem occurring because of PIDs resouce depletion seems yet unlikely since at most Linux boxes

$ cat /proc/sys/kernel/pid_max
32768

gives you 32x times the PIDs needed even for case script 1 and the processes/programs involved (i.e. sed , ping, etc) also seem unlikely to cause the inconstant results.

As mentioned by user @derobert the real issue behind the scripts failing was that the missing of the wait command, which means that after backgrounding the commands in the loop the end of the script and hence the shell caused all the child processes to be terminated.

Solution 2

This will do what you are expecting for:

#!/bin/bash
function_ping(){
    if ping -c 1 -w 5 $1 &>/dev/null; then 
        echo "UP: $1"
    else
        echo "DOWN $1" 
    fi
}
for ip in {1..254}; do
        function_ping 192.168.1.$ip &  
done
wait 

Save it as parallelping, and execute it.

It helps you? It can be converted into a big function, that can be used in a fast while loop too so you can use your imagination programming with it.

Note: You must use bash.

Comments

[Admin]

apt-get install fping :-/ (or nmap -sP)

[Admin]

@derobert I know nmap works. I just wanna try doing a ping sweep for this experiment.

[Admin]

Well, the right way to do this is to keep track of how many child processes you've got running, and keep to a sane number. Other than using utilities (e.g., parallel) to do it for you, you'd have to keep track of which children you've spawned (you get the pid from $!, wait for them to exit (and find out their exit code) from wait, etc. Personally, I switch to Perl at this point...

[MykelXIII]

It works, however I removed the & in the ping -c 1 -w 5 $1 & statement. Can you explain why there are 2 &? One in the function invocation and one in the call to ping? Also can I ask why subshells fail while putting functions to background don't?

[Luciano Andress Martini]

The & in the ping statement is helping for redirecting the ping output to /dev/null, it means that all standard output of the command needs to be redirected, so is nothing related to the other "&" that will run the function on background but to the redirect character (>), this works when this character '&' is leaning on '>' character. Your script is working very fine too, you need to put a wait after 'done' statement or the script will not wait the background processes to terminate. Guarantee that you will run it from bash and not from sh like 'sh thescript'.

[humanityANDpeace]

What is the crucial information of the answer? Is it to use wait as to wait for the termination of the backgrounded child processes, yes? Or was the crucial point the use of a shell function? As writen in the comment by @LucianoAndressMartini it seems to be the wait, yet this does not really become clear from the answer.

[MykelXIII]

--deleted contents--

[humanityANDpeace]

@MykelXIII :). being curious: what do you mean more precisely by "the script depleted resources"? how do you measure/test that? A resource could be the use of a PID (there are not unlimted process ids), memory, cputime, what did you mean?

[MykelXIII]

Sorry, I made a mistake on understanding your answer. It makes sense now. The first script actually used more resources. I am curious though, if my first script did not wait and still brought consistent results, does that mean that it was because of the child processes completing faster than the parent process?

[humanityANDpeace]

@MykelXIII My guess is this: In both scripts, because there is no wait ing to guarantee not to prematurely kill the spawned subprocesses, there is clearly a race..... The end is after the script went through the for loop... script 1 yields better results, because (a) each in script1's loop it takes longer (script1 : start pipe with 4 process VS script2: start a subshell) until stuff was put into background. Additionally the in (b) script2 ping is run only after the subshell was fork() and hence later than in script1 and it takes another if , [ and echo command to....

[humanityANDpeace]

... to finish, which is time that script2 does not provide as the loop finises (see (a) quickly)

[Luciano Andress Martini]

It is a old question, but you're right humanity, i clarified the question on the comment, because i realized that the original script is working, after writing the answer.