Script to email all matching files in a directory

Question

Trying to set up a bash script to automate sending data files. Desired end result is to check a specified directory once per week, make a list of all Excel files modified within the last 2 days, and email them to a designated recipient list.

I have sendmail configured and working (using a Gmail relay). I have mutt configured and working. In general the commands I am trying to send work if send directly from CLI (mail is sent and received, with attachments) but I get repeated failures when trying to call them from the script. Everything seems to stem from the fact that the directory and file names contain spaces. I can't change this - I don't have control over the naming of the files - but it seems as if this is the sticking point in my script.

Two main problems:

1. If I try to send all files in one go, mutt reports the attachment(s) cannot be attached:

Can't stat "/path\ to/file1.xls
/path\ to/file2.xls": No such file or directory

The newline char is being passed as part of the $FILES variable.

1. If I try to loop through the directory to send files one at a time (my desired outcome, as some files are fairly large) the script interprets spaces as delimiters, escaped or not - so /path/to/the\ files/file\ 1.xls is seen as 3 values (/path/to/the\, files/file\, 1.xls).

I did have the first half of the script working (mailing all files at once) but managed to break it trying to add the loop. Of course I didn't save the earlier working version. Tried using set ifs=$'\n' for the loop to get the delimiters correct, but when I have that in place mutt tells me the full path to the file is not a file, or that there was no recipient designated. It's a bit maddening.

Script:

#!/bin/bash
# This file should send an email to specified recipients with data files for the week attached.

# Set reply-to address for Mutt
export REPLYTO="[email protected]"

# Replace with space separated email address of all recipients
EMAILS="[email protected] [email protected]"

# Get today's date for subject
# Date is in YYYY-MM-DD format
TODAY=$(date +%F)

# Set the message body
MBODY="Sending this week's data files.\n"

# Set the starting directory
# Don't bother escaping it, this is fixed in FILES variable below
DIR="/home/user/path to files"

# Get the list of files to send
FILES=$(find "$DIR" -type f -mtime -2 | sed 's/ /\\ /g' | grep ".xls")

# Check to see if we found any files or not
if [ -z "$FILES" ]; then
    MBODY="No matching files added or modified within last 2 days. No files sent.\n"
    echo "$MBODY" | mutt -s "Data files for $TODAY" $EMAILS
fi

# Send all files in a single email
echo $MBODY | mutt -s "Data files for $TODAY" -a "$FILES" -- $EMAILS

For sending files separately, I tried the following instead of the last 2 lines above:

# Cycle through FILES array and send each file individually
for datafile in ${FILES[*]}
do
   set IFS=$'\n\t'
   echo $MBODY | mutt -s \"Data files for $TODAY\" -a $datafile -- $EMAILS 
   unset $IFS
done

Any help? I've been stuck on this for a while now. I'm not married to using mutt, or even bash for that matter, if this is easier done in another language.

Solution 1

set "/home/user/path to files/"*.xls
for f do [ "$f" -nt "$two_day_old_file" ] && set "$@" "$f" ; shift ; done
touch "$two_day_old_file"
echo $MBODY | mutt -s "Data files for $TODAY" -a "$@" -- $EMAILS    

To mail them one at a time change the echo line to:

for mailf do echo "$MBODY" | 
    mutt -s "Data files for $TODAY" -a "$mailf" -- $EMAILS
done

Would probably work, but your real problem lies here:

...
set IFS=...
...

This doesn't affect the value of the internal field separator at all, but rather it assigns the value IFS=... to the first positional parameter, or $1. $IFS remains valued at whatever it was before you set $1. You just need to do:

IFS='
   ' 

Or...

IFS=${IFS# } 

...if $IFS is set to the default value, which it must be if this is an executable script and you have not altered $IFS anywhere else in the script.

Solution 2

The problem is $FILE is not an array but you're accessing it like it is

for datafile in ${FILES[*]}

It's just returning a giant string, hence all the files at once.

To work around this, append a newline to each file in $FILE then in the loop use echo -e to return each line indivivually

# Get the list of files to send
FILES=$(find "$DIR" -type f -mtime -2 | sed 's/ /\\ /g' | grep ".xls" | sed '/.xls$/ a\\\n')

then

for datafile in $(echo -e $FILES)
do
   echo $MBODY | mutt -s \"Data files for $TODAY\" -a "$datafile" -- $EMAILS 
done

Additionally, adding double quotes to $datafile should fix the problem with whitespaces in the filenames

Solution 3

Seem to have figured out a working method. Avoiding $IFS and not trying to handle the entire directory list as a string or array - working on each entry as it is found. Replaced last loop with the following:

find "$DIR" -type f -mtime -2 -name '*.xls*' -exec sh -c '
  for file do
    # Check to see if we found any files or not
    if [ -z "$file" ]; then
        echo "$NOBODY" | mutt -s "Data files for $TODAY" $EMAILS
    # If we found files, then email them
    else
        echo "$MBODY" |  mutt -s "Data files for $TODAY" -a "$file" -- $EMAILS
    fi
  done
' sh {} +

where $NOBODY is a different message body stating no matching files were found. Using "*.xlsx*" to ensure both .xls and .xlsx files will be matched. This worked, but the variables declared above were not getting passed into the loop. I prefaced each with an export command, after which the above does the trick. (If there's a more elegant way to get the variables into the loop, I didn't find it. 99% of the discussions on loops and variables are about how to get them back OUT of the loop, not INTO it.)

Comments

[Warwick]

I don't know how to fix this either, but could you perhaps have a directory whose name doesn't contain spaces, copy the file into that directory, zip the directory, and send the zipped directory? The file name should then be preserved when the zip file is extracted. You could then clear the directory and copy the next file into it. Just looking at other strategies although I am sure you will get the answer you want shortly. I'll be interested in knowing it too.

[mikeserv]

set IFS= probably does not do what you want it to do.

[dr.nixon]

This does work - however it sends all files at once, which ends up being too large (some files are ~10 mb). It also returns an error (cannot touch `': No such file or directory) and grabs everything rather than just files modified within the last 2 days. I'd like to keep the file selection limits (anything within 2 days) and send each separately. This is helping, but I'm still stuck trying to get it to send one at a time.

[mikeserv]

@dr.nixon I'm not very familiar with mutt, but once you have the file list in $@ you can just do: for mailfile do something to "${mailfile}"; done. So maybe for mailfile do echo $MBODY | mutt -s "Data files for $TODAY" -a "$mailfile" -- $EMAILS ; done

[mikeserv]

as far as I can tell... the problem is not lack of quoting, but rather it is too much quoting: /"/path\ to/file1.xls /path\ to/file2.xls" - that's two arguments strung together in quotes, and mutt can't understand how to stat two files as one - it can't delimit between one file and another. That's why I remarked on $IFS - and it is, as near as I figure, also the reason IFS is in there at all.

[Chad K]

@mikeserv that is a problem but it's not the problem. The real problem is OP wants to email each file individually but his code is attaching all the files at once

[mikeserv]

I know - I read his comment on my answer. he just needs a for loop - and to ensure the argument delimiters remain in tact. you cam't separate fields without a field separator. If you do $(echo files) it's going to break. Just set them in the shell array: set /path/to/files; for f do something; done` or for f in /path/to/files; do something; done. This entire situation surfaced because an argument list was treated like a string - doing it again is no solution.

[mikeserv]

Or if you like find do -exec mutt - he'll probably accept it. But the string parsing is asking for trouble.

[dr.nixon]

find -exec do mutt is the trick. See answer.

[mikeserv]

If you want the vars in the loop, consider doing find ... -exec sh -c 'SCRIPT' {} $var1 $var2 \; - i think that works anyway... Gotta check it - never tried. Anyway, if it does your mail files will be in $0 and vars 1/2 in $1 $2. Youll get one mail file per loop. Nice answer, by the way.

[Chad K]

@mikeserv but echo -e will interpret the newline escapes and the loop returns each filename seperatly instead of the giant blob from find. HOWEVER, I didn't realize what set /path/to/files for blah does and I like that much better, +1. I'm actually shocked at how hard it was to find anything documented about how that works.

[mikeserv]

Dammit. Stupid android deleted my comment - anyway, subshell just provides standard out like echo * | read : set manipulates arguments - you get a listed array. Its handy. Google rich's posix sh tricks or man set or look at my answers - im a weird one.

[Chad K]

yea, it's slick. I had no idea how to create an array without ( adding everything in manu-freaking-ly) or looping like an idiot, hence the dirty workaround. I shall now go google in shame

[mikeserv]

This one might be my best but Rich's tricks is way better. I tend to treat the shell like a functional language - one array per function, and thats all i want. Args, do, done. I hate names. Oh, and it also makes handing arguments over a pipe robust sh -c 'set stuff; (process "$@"; echo '$1') | function "$@"

[Chad K]

I hate names I can see that. That's why set/the for loop threw me off. I'll find you in chat tomorrow