Simple queuing system?
Solution 1
There's a standard batch
command that does more or less what you're after. More precisely, batch
executes the jobs when the system load is not too high, one at a time (so it doesn't do any parallelization). The batch
command is part of the at
package.
echo 'command1 --foo=bar' | batch
echo 'command2 "$(wibble)"' | batch
at -q b -l # on many OSes, a slightly shorter synonym is: atq -q b
at -q b -r 1234 # Unschedule a pending task (atq gives the task ID)
Solution 2
Another solution is to use lpd
, and create a custom "print driver" that runs your jobs. A friend helped me work this out when I had a similar request. Make a script like this, and put it in /tmp/batch.sh
:
#!/bin/bash
TMPFILE=$(mktemp /tmp/XXXX)
exec <"$6"
cat - > $TMPFILE
chmod a+x $TMPFILE
$TMPFILE
rm -f $TMPFILE
Then run:
lpadmin -p batch1 -E -P /tmp/batch.sh
That starts a queue, and you can create more by using other names instead of batch1. Add a job with:
lp -d batch1 /path/to/jobscript
Manage jobs with lpq
, lprm
, and lpstat
. If you want more flexibility with passing arguments to your jobs, you can make the batch.sh script fancier.
(I tried batch
before going down this route, but either it doesn't work as a queue on OSX, or I was using it wrong.)
Solution 3
I notice this question is several years old, so it may not help the original poster, but it may help someone else.
First: "task spooler" is the answer. It's pretty powerful and most Linux distributions and Homebrew have it.
But a lot of the servers I use, I can't install arbitrary packages without a lot of hassle, so I need something that is ideally pure bash (or perl, or such).
After struggling with this for a while, I came up with a pure bash implementation that appears to work fine so far. You can find it at https://github.com/sitaramc/notes/blob/master/bq, with documentation.
It's just one bash script so installation is trivial. However, it punts your second and third requirements (but it should be trivial to implement those too).
The script is liberally commented and you should be able to review it in a few minutes if you wish to.
Solution 4
There are lots of queuing systems, but the are frequently very specialized.
You might look into the at
scheduler. It's like cron
in some ways but it is setup more like a queue for one time jobs than for repeat jobs. It can "schedule" things on criteria other than time, such as system load or sequence of jobs.
Your favorite distro will almost certainly have packages for it.
Related videos on Youtube
dagnelies
Updated on September 18, 2022Comments
-
dagnelies over 1 year
Given a commodity PC, we would like to use it to execute some tasks in the background round the clock.
Basically, we would like to have commands like:
add-task *insert command here* list-tasks remove-task(s)
The added tasks should simply be put in a queue and executed one after another in the background (keeping running after logout of the shell).
Is there any simple script/program that does this?
-
Joe Fusion over 10 yearsI realize that @arnaud specified "commodity PC," which means it's probably not OSX, where I tested this solution. However, this should be portable, and it's much more flexible than
batch
. -
rsaw about 8 yearsIt's wrong to say that the
batch
command "doesn't do any parallelization". There's usually a default 60sec delay between starting one job and starting the next; however, there's nothing to make the next job wait for the first to finish --atd
will happily kick off jobs from the batch queue as soon as the value set by the-b
option has elapsed (seeatd
man page). -
Thiago Macedo over 6 yearsThis is possible one of the most genuine hacks I've ever seen.
-
ergosys almost 5 yearsBatch doesn't strictly serialize the tasks, so if you have a single shared resource, @sitaram 's answer (task spooler) should work better.
-
ergosys almost 5 yearsdebian package is task-spooler, executable is "tsp" instead of "ts" which collided with something else. I like it also because you can reorder tasks that are in the queue, you can't do that with at/batch.
-
wdkrnls almost 5 yearsSadly, this no longer appears to work. github.com/apple/cups/commit/…
-
rcoup over 2 yearsOne minor but important difference is that bq starts the tasks as a child of the worker process, whereas tsp starts it as a background child of the enqueuing process and the server process just monitors it.
-
wolfram77 over 2 yearsThis is a CPU/GPU task spooler (job queue) which worked for me: github.com/justanhduc/task-spooler.