lsf bkill all PEND jobs without killing RUN jobs

Question

I'v lots bjobs running on lsf, jobs have two status, RUN and PEND. And I want to kill all bjobs with PEND status, how to do that use script? A hard-coded way I think is saving them in a file then parse every line to get the status and key. If the STAT is PEND then pass the key to bkill $key. But this is very complicated, is there any bkill function that can directly do this or a non hard-coded way to kill jobs with a specific status or name?

Solution 1

bjobs can list just the pending jobs with -p. It would be nice if bkill -p would also filter jobs, so bkill -p 0 would kill all the user's pending jobs.

The best I can think of is some shell magic. While not perfect, it should cover many cases. e.g.,

bkill `bjobs -p -o id -noheader | tr '\n' ' '`

bjobs -p -o id -noheader lists the job ids of the user's pending jobs. tr will put it in the format that bkill expects. One potential problem is that if the job list is too long then the max command line length or max number of command line arguments will have been exceeded, so bash will complain.

There's a small race condition here. A job could start between the query and the kill.

Solution 2

An alternative solution (also using shell filters as the previous answer, but without the risk of exceeding the no of allowed shell command arguments):

bjobs -w | grep 'PEND' | awk '{print $1}' | xargs bkill

Basically check for running jobs (-w wide format, no truncating), use grep to filter PENDing job, then select jobIDs with awk, and pass them to bkill via xargs (xargs breaks the no of arguments passed to bkill, avoiding "Argument list too long" kind of errors).

Comments

[Matthias Schuchardt]

For brevity, one could use id instead of jobid. Also -noheader instead of the | grep -v ^JOBID would be shorter, more speaking and less dependent on how LSF names its column headers.