Taskset not working over a range of cores in isolcpus

Question

To preface I'm using Debian Wheezy with kernel 3.2 on an AMD64 chipset. My machine has two Xeon E5-2690 cores. I set up the boot parameters so that all the cores on one CPU are dedicated to a single process. To do this I've set isolcpus=8,9,10,11,12,13,14,15 in grub.

So far, so good. Now let's say I want to use the isolated CPUs for a given command, to be simple I'll just use a simple infinite loop:

$ taskset -c 8-15 bash -c 'while true ; do echo hello >/dev/null; done' &

So far so good, top shows that core 8 spins up to near 100% utilization. Now let's say I launch that command again:

$ taskset -c 8-15 bash -c 'while true ; do echo hello >/dev/null; done' &

Now top shows that cores 9-15 remain idle and the two processes are sharing core 8. If instead I do this:

$ taskset -c 8 bash -c 'while true ; do echo hello >/dev/null; done' &

$ taskset -c 9 bash -c 'while true ; do echo hello >/dev/null; done' &

Cores 8 and 9 each get 100% utilization as they should. This only applies to isolcpus because the same taskset with cores 1-7 properly spreads the processes over the relevant cores. Furthermore "taskset -p" shows that the affinity mask for the 8-15 processes are set correctly. It appears the kernel scheduler refuses to use anything but the lowest core specified on an isolcpus affinity mask.

Now normally this wouldn't be a big deal with my above examples, just specify individual cores for each process. However I want to run a highly multithreaded application on the dedicated CPU. I want to specify the core set and have the thread pool automatically use, without having to individually reset the processor affinity for each individual thread that's spawned.

Does anyone have any idea how to get the scheduler to give me more than one core from the isolcpu set?

Solution

After a day of frustration I've determined a solution. This behavior seems to be an artifact of the default kernel scheduler algorithm (SCHED_OTHER for this distro/kernel). Changing the process to a different algorithm eliminates the problem, isolcpus are adequately utilized across the processes/threads.

I ended up using SCHED_RR, but I also tested SCHED_FIFO and SCHED_IDLE both of which seem to work. The process can be launched with the alternative algorithm by use of the chrt utility:

$ sudo chrt -r 1 [command]

(If you want to run as non-root you can use the setcap utility to enable CAP_SYS_NICE on the binary file related to the command)

Comments

[Admin]

can you try using a multithread program? bash isn't multithread

[Admin]

read this linuxtopia.org/online_books/linux_kernel/kernel_configuratio‌​n/…, this exclude a cpus from kernel scheduler, but after you exclude cpus you want to run process on the excluded cpus ?

[Barry NL]

Although tasksetting the affinity to cores 0,1 my java application only utilized the first core. 'sudo chrt -r 1 [command]' solved my problem too.

[Anton9988]

Faced same issue, however if the process using RR on isolated cores is perpetually running or requires additional process to run on same core it won't be possible. In this scenario only one process will occupy core and will free it only once it's completely done. Any idea on how to allow multiple processes to run on isolated cores?