Whole one core dedicated to single process

Question

Is there any way in Linux to assign one CPU core to a particular given process and there should not be any other processes or interrupt handlers to be scheduled on this core?

I have read about process affinity in Linux Binding Processes to CPUs using the taskset utility but that's not solving my problem because it just try to affine the given process to that core but it is possible that other processes may be scheduled on this core and this is what I want to avoid.

Should we change the kernel code for scheduling?

Solution 1

Yes there is. In fact, there are two separate ways to do it :-)

Right now, the best way to accomplish what you want is to do the following:

1. Add the parameter isolcpus=[cpu_number] to the Linux kernel command line from the boot loader during boot. This will instruct the Linux scheduler not to run any regular tasks on that CPU unless specifically requested using cpu affinity.

2. Use IRQ affinity to set other CPUs to handle all interrupts so that your isolated CPU will not receive any interrupts.

3. Use CPU affinity to fix your specific task to the isolated CPU.

This will give you the best that Linux can provide with regard to CPU isolation without out-of-tree and in-development patches.

Your task will still get interrupted from time to time by Linux code, including other tasks - such as the timer tick interrupt and the scheduler code, IPIs from other CPUs and stuff like work queue kernel threads, although the interruption should be quite minimal.

For an (almost) complete list of interruption sources, check out my page at https://github.com/gby/linux/wiki

The alternative method is to use cpusets which is way more elegant and dynamic but suffers from some weaknesses at this point in time (no migration of timers for example) which makes me recommend the old, crude but effective isolcpus parameter.

Note that work is currently being done by the Linux community to address all these issues and more to give even better isolation.

Solution 2

There is Redhat article talking about it. It modifies the boot parameter isolcpus.

And an old article written by Robert Love. And there is solution in that article.

All of a process' children receive the same CPU affinity mask as their parent.

Then, all we need to do is have init bind itself to one processor. All other processes, by nature of init being the root of the process tree and thus the superparent of all processes, are then likewise bound to the one processor.

Solution 3

Dedicate a Whole CPU Core to a Particular Program

While taskset allows a particular program to be assigned to certain CPUs, that does not mean that no other programs or processes will be scheduled on those CPUs. If you want to prevent this and dedicate a whole CPU core to a particular program, you can use "isolcpus" kernel parameter, which allows you to reserve the CPU core during boot.

Add the kernel parameter "isolcpus=" to the boot loader during boot or GRUB configuration file. Then the Linux scheduler will not schedule any regular process on the reserved CPU core(s), unless specifically requested with taskset. For example, to reserve CPU cores 0 and 1, add "isolcpus=0,1" kernel parameter. Upon boot, then use taskset to safely assign the reserved CPU cores to your program.

Source(s)

1. http://xmodulo.com/2013/10/run-program-process-specific-cpu-cores-linux.html
2. http://www.linuxtopia.org/online_books/linux_kernel/kernel_configuration/re46.html

Solution 4

Even if you follow the steps in gby's answer, kernel tasks are executed on the isolated CPU core. Work is underway in the linux RT_PREEMPT real time project to improve this. So if you are not using a bleeding edge real time kernel from RP_PREEMPT, it might not be possible to completely isolate a CPU core.

View more solutions

Comments

[Maxim Egorushkin]

I read that full CPU isolation is not implemented yet: lwn.net/Articles/520704 (scroll down to "CPU isolation").

[louxiu]

Can not add the second link because the domain is number.:-(

[Hasturkun]

Fixed the link, article originally from Linux Journal

[Sergei Nikulov]

That's exactly what I mean. You just give a hint to scheduler to run process on specific core/processor. But it is not a way to exclusively allocate core/processor for specific proress.

[akp]

yes u are right till now 100% isolation is not possible!!! as some APIC timer interrupt may execute on the given core,etc. but gby's answer give considerable isolation.

[Cedomir Segulja]

Is it still true that isolcpus gives better isolation than cpusets, even when --kthread=on option is provided?

[Kho Dam]

@CedomirSegulja I believe it does although the difference narrows over time. For example, if a kernel module registered a timer on a CPU and that timer is self registering (e.g. it registers itself again when it elapses), currently there is no simple way to move it off the would-be dedicated CPU. With isolcpus the chance that such a timer would be registered in the first place is much lower.

[MonoThreaded]

A Redhat subscription is required for this link. Could you please post its content?

[Mark]

@gby, does isolcpus parameter accept physical core id or logical? I've read kernel.org/doc/Documentation/kernel-parameters.txt however it's not 100% clear to me. Specifically I'm interested about Hyper-Threading mode, which gives 2 logical cores per 1 physical on my Xeon CPU, and what should I specify in icolspus. Thanks.

[Kho Dam]

@Mark logical. Same ID that sched_setaffinity accepts

[Mark]

@gby, thanks for response. Am I right assuming that with hyper-threading mode disabled, LogicalID == PhysID of the CPU ?

[Gilgamesz]

"Use IRQ affinity to set other CPUs" @gby, how to do it?

[Kho Dam]

@Gilgamesz the kernel has a good doc on it: kernel.org/doc/Documentation/IRQ-affinity.txt