How to iterate through a comma-separated list and execute a command for each entry

Question

I have been give a command which outputs a comma-separated list for autosys jobs in the variable $all_jobs:

box=box-of-jobs;all_jobs=$(jobscout -box $box | egrep "^\w+" | tr '\n' ','  | sed s/.$//); 

I want to call a sendevent command for each item in the list:

sendevent -verbose -S NYT -E JOB_OFF_HOLD -J $job --owner me

So for example if $all_jobs evaluated job1,job2,job3

I want to call

sendevent -verbose -S NYT -E JOB_OFF_HOLD -J job3 --owner me
sendevent -verbose -S NYT -E JOB_OFF_HOLD -J job2 --owner me
sendevent -verbose -S NYT -E JOB_OFF_HOLD -J job1 --owner me

I'm sure I could write a ksh script to loop through, but I know these things can often be written much faster using awk/sed, both of which I'm not very familiar with so it's beyond my megre skills. I'm using ksh (rather than bsh).

Solution 1

If you're sure that the fields between the commas do not contain any whitespaces than you could do something like this:

for job in $(echo $all_jobs | tr "," " "); do
    sendevent -verbose -S NYT -E JOB_OFF_HOLD -J "$job" --owner me
done

If you need something more robust, take a look at the tools needed to deal with CSV files under Unix.

Solution 2

Splitting a string at a particular character is built into the shell at a very deep level: if you write $var without any surrounding quotes, then it is expanded as follows:

1. Take the value of the var variable.
2. Split this value into a list of fields. The field separator is any character in the value of the IFS variable. If IFS contains whitespace characters, then consecutive separators are considered as one; for non-whitespace characters, consecutive separators result in empty fields.
3. Perform globbing, i.e. interpret each resulting field as a file name wildcard pattern. For each pattern that matches one or more files, replace it by the list of matching files.

The same happens for a command substitution $(somecommand), except that step 1 is “collect the output from running somecommand and strip all newlines at the end”.

To avoid all this rigmarole and just get the exact value of the variable, or the output of the command minus final newlines, be sure to put the variable substitution or command substitution between double quotes: "$foo", "$(foo)".

To split the result of a command susbtitution at comma characters, set IFS=, and leave the substitution unprotected. You need to do one more thing: turn off globbing, with set -f (and restore it afterwards with set +f.

all_jobs=$(…)
set -f; IFS=,
for job in $all_jobs; do
  sendevent -verbose -S NYT -E JOB_OFF_HOLD -J "$job" --owner me
done
set =f; unset IFS

Comments

[Ian]

Thank you - this worked perfectly (plus thanks for the further reading link). It doesn't need to be particularly robust - it's just a labour-saving device for setting things up - not production code. Thanks again!

[Ian]

Thanks - I tried to use IFS when I was googling for a solution before posting the question but I think I had the syntax slightly wrong - thanks for the example!