GNU Parallel: immediately display job stderr/stdout one-at-a-time by jobs order

Question

I know that GNU Parallel buffers std/stderr because it doesn't want jobs output to be mangled, but if I run my jobs with parallel do_something ::: task_1 task_2 task_3, is there anyway for task_1's output to be displayed immediately, then after task_1 finishes, task_2's up to its current output, etc.

If Parallel cannot solve this problem, is there any other similar program that could?

Solution 1

From version 20160422 you can do:

parallel -k --lb do_something ::: task_1 task_2 task_3

Solution 2

parallel actually makes no reservations about making output come in order. It just so happens that jobs are typically small enough and evenly split in cpu time to come out in order. You notice it more when you have a lot of jobs running or their task duration differs a lot.

By default parallel will only run the same amount of jobs as your CPU has cores. On most laptops and desktops that's 2-4 which means it's only running a couple of jobs at a time. You can increase that with -j.

Here's an example to demonstrate job order does not output in the order it was submitted.

seq 20 | parallel -j 20 'sleep $[RANDOM % 20]; echo '

output on my system was (yours will likely be different)

7
3
13
20
8
16
2
4
18
17
1
5
9
14
12
6
10
19
11
15

seq 20 is a command that will output numbers 1-20. I pipe that to parallel and then tell it to run 20 simultaneous jobs to make sure they all start at the same time. 'sleep $[\[RANDOM][1] % 20]; is using sleep plus a zsh parameter that will return a random number between 1 and 20. Each job will sleep this random amount and then echo. Once the job echos you'll immediately get the output from parallel.

You could also do something similar with parallel --shuf which will shuffle the job order.

Comments

[Hai Luong Dong]

Actually, it can with -k/--keep-order (I used parallel version 20141022) seq 20 | parallel -k -j2000% 'sleep $[RANDOM % 20]; echo ' 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

[Hai Luong Dong]

I've just installed it and it works exactly how I wanted. Thank you very much!

[zyxue]

There are --ungroup as well if you don't care about the order at all.

[Ole Tange]

From version 20170822 --lb is not much slower than --ungroup.