Catching the exception thrown by python script in shell script

Question

I am having a shell script which opens a file and pass it to python script for processing it. So if there is any issue with the file ( e.g. the file content is not in the format required for the successful execution for python script), the python script throws an exception. Since my objective it to process N files using python script. I need to know which is the file causing the script to break. I read on how to catch the exception in thrown by command execution. http://linuxcommand.org/wss0150.php . But in my case its the python script which throws the exception and I need to know in shell script what the exception has been thrown. Can anyone help me how can I proceed with this?

Below is the code snippet :

#!/bin/bash
yesterday=$(date --date "yesterday" "+%y%m%d")
ftoday=$(date --date "today" "+%m-%d-%Y")
year=$(date "+%Y")
fileList=$(find C:/logdata/$year/$ftoday/ -iname "*"$yesterday".log")
for var in $fileList
do
echo -e "\n START Processing File : $var" >> shelltestlog.txt
cat $var| ./scriptA.py 
echo -e "\n END Processing File : $var" >> shelltestlog.txt
done

Solution 1

If your python script returns an non-zero error level whenever it gets an exception, you can use || { } to log messages:

./scriptA.py < "$file" || {
    printf "\n Python script scriptA.py failed with file \"%s\".\n" "$file" >> shelltestlog.txt
} 

I actually tried to simplify your code first:

#!/bin/bash

yesterday=$(date --date "yesterday" "+%y%m%d")
ftoday=$(date --date "today" "+%m-%d-%Y")
year=$(date "+%Y")

readarray -t filesList < <(find C:/logdata/$year/$ftoday/ -iname "*"$yesterday".log")

for file in "${filesList[@]}"; do
    printf "\n START Processing File : %s\n" "$file" >> shelltestlog.txt
    ./scriptA.py < "$file" || {
        printf "\n Python script scriptA.py failed with file \"%s\".\n" "$file" >> shelltestlog.txt
    }
    printf "\n END Processing File : %s\n" "$file" >> shelltestlog.txt
done

Solution 2

An uncaught exception will produce a traceback printed to standard error. The best you can do is capture that, and try to parse it.

if ! ./scriptA.py < "$var" 2> stderr.txt; then
  # Parse stderr.txt to see what exception was raised.
fi

Solution 3

You should rewrite the shell script in Python and merge it into the existing Python script. The stuff that you are currently doing with the date command, can be done with the datetime and time modules. The stuff you are doing with find can be done with os.walk and fnmatch.fnmatch.

Here's an outline:

#! /usr/bin/python

def scriptA(file_to_process):
    # the code currently in scriptA goes here;
    # modify it to read from `file_to_process` rather than `sys.stdin`

def get_file_list():
    # your bash logic to construct the file list goes here,
    # converted to Python as suggested above

def main():
   for filename in get_file_list():
       sys.stderr.write("Processing {}...\n".format(filename))
       try:
           scriptA(open(filename, "rt"))

       except SomeExceptionType as e:
           # handle exception...

       except SomeOtherExceptionType as e:
           # handle other kind of exception...

       sys.stderr.write("Done processing {}.\n".format(filename))

main()

Solution 4

You can play some redirection games:

# open file descriptor 3, pointing to the same destination as stdout
exec 3>&1

# run the script, 
# python's stderr goes to stdout so it can be captured in a variable
# python's stdout goes to file descriptor 3, displayed on the terminal
python_stderr=$( ./scriptA.py < "$var" 2>&1 1>&3 )

# close the file descriptor
exec 3>&-

Then you can examine $python_stderr for patterns

case $python_stderr in
    "") echo "nothing on stderr" ;;
    *foo*) handle_foo_error ;;
    *bar*) handle_bar_error ;;
    *baz*) handle_baz_error ;;
esac

View more solutions