How to check the status of bash shell script while executing from Python script?

Question

I have a simple Python script which will execute a shell script using subprocess module in Python.

Below is my Python shell script which is calling testing.sh shell script and it works fine.

import os
import json
import subprocess

jsonData = '{"pp": [0,3,5,7,9]}'
jj = json.loads(jsonData)

os.putenv( 'jj3', ' '.join( str(v) for v in jj['pp']  ) )

print "start"
proc = subprocess.Popen('testing.sh', stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(stdout, stderr) = proc.communicate()
if stderr:
   print "Shell script gave some error"
   print stderr
else:
   print stdout
   print "end" # Shell script ran fine.

And below is my testing.sh shell script -

#!/bin/bash

dir1=some_directory
dir2=some_directory

length1=some_number
length2=some_number

if [ "$dir1" = "$dir2" ] && [ "$length1" -gt 0 ] && [ "$length2" -gt 0 ]
then
    for el in $jj3
    do
        scp david@machineB:/data/be_t1_snapshot/20131215/t1_"$el"_5.data /data01/primary/. || scp david@machineC:/data/be_t1_snapshot/20131215/t1_"$el"_5.data /data01/primary/.
    done        
fi

What my above shell script does is, it will copy the files from machineB OR machineC to machineA. If the files are not there in machineB then it should be there in machineC always. So it will try to copy the files from machineB to machineA but if the files are not there in machineB then it will try to copy the files from machineC to machineA.

Now my above Python script (which I am running from machineA) tries to execute my above shell script and see whether my script got executed successfully or not. As I am storing the stderr of my shell script and stdout as well.

Problem Statement:-

With the above approach there is one problem that I am seeing. As I mentioned if the files are not there in machineB, then it will try to copy the files from machineC to machineA. So whenever I run my Python script which calls my shell script, what happens is that, some files are not there in machineB so it throws some exception and then it tries copying the files from machineC but when the call comes back to Python script, it always go inside if stderr: block as there was an error while copying the files from machineB (which is not what I want) and end print statement doesn't gets printed out.

So the question is if the files are not there in machineB, it will try copying the files from machineC and if the file is there is machineC and got successfully copied to machineA without any error, then I want to call that as a success instead of failure. In current scenario what is happening, if the files are not there in machineB and got successfully copied from machineC then it still counts as a failure and end print statement doesn't gets printed out.

I also want to see whether my shell script has any problem while executing from Python. If yes, then I don't want to print end statement.

How should I overcome this problem?

UPDATE:-

What will happen with the below shell script? As the first for loop will failed because david is not a linux command, but my scp command works fine. So still I will see status code as not equal to 0?

for i in $( david ); do
    echo item: $i
done

dir1=some_directory
dir2=some_directory

length1=some_number
length2=some_number

if [ "$dir1" = "$dir2" ] && [ "$length1" -gt 0 ] && [ "$length2" -gt 0 ]
then
    for el in $jj3
    do
        scp david@machineB:/data/be_t1_snapshot/20131215/t1_"$el"_5.data /data01/primary/. || scp david@machineC:/data/be_t1_snapshot/20131215/t1_"$el"_5.data /data01/primary/.
    done        
fi

Solution

From Python Subprocess Doc :

Popen.returncode

The child return code, set by poll() and wait() (and indirectly by communicate()). A None value indicates that the process hasn’t terminated yet.
A negative value -N indicates that the child was terminated by signal N (Unix only).

So you can check exit status using :

proc.returncode

but it's bit ironic thing is zero 0 means False in Python.

Example :

if (var == False) :
    # this will execute if var is False or 0 (or 0.0, 0L, 0j)

Read Python Pitfalls

Comments

[Mathias Begert]

Get the bash script to use its exit code to explicitly signal success or failure to the python script, and use subprocess.check_output instead of subprocess.communicate. All the best

[arsenal]

Can you provide an example basis on my above shell script?

[Keith]

Any reason why you can't just do it all in Python?

[arsenal]

Thanks Rahul, But if I am using set -o in my shell script. And if it tries to copy the file from machineB and if the files is not there in machineB so it throws an exception but that file will be there in machineC so it will copy the files from machineC. But if I am using set -o then it will stop the shell script as soon as it tries to copy the file from machineB which is also not what I want?

[Rahul Patil]

@TechGeeky yes... I miss that point.. sorry I read slowly ... give me some time..

[Rahul Patil]

@TechGeeky Why you not do all things in python ?

[arsenal]

That's fine.. I cannot do all these things in Python. This is the requirement that has been setup so need to follow that..

[arsenal]

Also, I updated my question with another simple shell script. What will happen in that scenario? I will see status code not equal to zero if any of the command in my shell script failed or only the last command?

[Rahul Patil]

any one, because you have used ||

[arsenal]

so if any one of them (scp) failed, I will still see error code as not equal to zero?

[Rahul Patil]

@TechGeeky you just want to exclude error of your shell script or what you exactly want to archive from Python script

[arsenal]

let us continue this discussion in chat