How to ban hardware interrupts with IRQBALANCE_BANNED_CPUS on Ubuntu?

Question

I would like to ban interrupts from certain CPUs. I heard about the IRQBALANCE_BANNED_CPUS option. I see irqbalance is running in the background of my machine. Where do I go to edit and how do I configure that option? For example, I want to exclude cpus 2,3,4,5 from interrupts. The argument descriptor is:

Provides a mask of cpus which irqbalance should ignore and never assign interrupts to

What does it mean by a mask? And where do I configure irqbalance with that option?

EDIT1: How to know that my configuration is in effect, in other words that my cpu is receiving NO interrupts? I am checking /proc/interrupts but some numbers are increasing there.

EDIT2: Now I booted my machine with IRQBALANCE_BANNED_CPUS=3e so only CPU 0 is NOT banned from interrupts. So I should expect to see cpo0 receiving a lot of interrupts and the other cpus not receiving interrupts, right? Here is my /proc/interrupts. The lines in bold are changing for ALL cpus. Lines 22, 24, 35 and LOC are changing.

            CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       
   0:         26          0          0          0          0          0   IO-APIC-edge      timer
   1:          2          0          0          0          0          0   IO-APIC-edge      i8042
   6:          3          0          0          0          0          0   IO-APIC-edge      floppy
   8:          1          0          0          0          0          0   IO-APIC-edge      rtc0
   9:          0          0          0          0          0          0   IO-APIC-fasteoi   acpi
  12:          4          0          0          0          0          0   IO-APIC-edge      i8042
  14:      13556          0          0          0          0          0   IO-APIC-edge      ata_piix
  15:          0          0          0          0          0          0   IO-APIC-edge      ata_piix
  18:          0          0          0          0          0          0   IO-APIC-fasteoi   ata_piix
  19:          2          0          0          0          0          0   IO-APIC-fasteoi   ohci1394
  20:          3          0          0          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb2, uhci_hcd:usb3, uhci_hcd:usb6
  21:        197        635         39          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb4, uhci_hcd:usb7, HDA Intel
  22:        344       3506          0        702          0          0   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb5, uhci_hcd:usb8
  24:        162         48          0          0          0          0   IO-APIC-fasteoi   nvidia
  35:        174          0         47          0          0          0   IO-APIC-fasteoi   nvidia
  53:       3517          0          0          0          0          0   PCI-MSI-edge      eth0
 NMI:          0          0          0          0          0          0   Non-maskable interrupts
 LOC:      11007       8840       6480       5652       4272       3046   Local timer interrupts
 SPU:          0          0          0          0          0          0   Spurious interrupts
 PMI:          0          0          0          0          0          0   Performance monitoring interrupts
 PND:          0          0          0          0          0          0   Performance pending work
 RES:        292        169        217        125        122        126   Rescheduling interrupts
 CAL:         86        280        254        292        293        291   Function call interrupts
 TLB:       1147       1031       1348        616        177        322   TLB shootdowns
 TRM:          0          0          0          0          0          0   Thermal event interrupts
 THR:          0          0          0          0          0          0   Threshold APIC interrupts
 MCE:          0          0          0          0          0          0   Machine check exceptions
 MCP:          2          2          2          2          2          2   Machine check polls
 ERR:          5
 MIS:          0

EDIT3: It looks like IRQBALANCE_BANNED_CPUS option is completely IGNORED on Ubuntu. I tried rebooting my machine with 1, 3e and got interrupts all over. Just when I disable irqbalance by setting ENABLED=0 that I get a clean /proc/interrupts just on cpu0 and no other cpu.

Solution 1

You set IRQBALANCE_BANNED_CPUS in /etc/default/irqbalance. I found this by looking in /etc/init.d/irqbalance. But what are the valid values for that setting? From Red Hat's man page:

This is a hex mask without the leading '0x', on systems with large numbers of processors each group of eight hex digits is sepearated ba a comma ','. i.e. export IRQBALANCE_BANNED_CPUS=fc0 would prevent irqbalance from assigning irqs to the 7th-12th cpus (cpu6-cpu11) or export IRQBALANCE_BANNED_CPUS=ff000000,00000001 would prevent irqbalance from assigning irqs to the 1st (cpu0) and 57th-64th cpus (cpu56-cpu63).

The concept of a mask is explained on wikipedia. Read that, then come back. Let's break down Red Hat's first example. The number that is written as fc0 in hexadecimal is written as 111111000000 in binary. Scanning right to left (i.e. from the least significant bit to the most significant bit), there are six zeroes. This means the 1st-5th cpus (cpu0-cpu5) can be assigned interrupts. Then, there are six ones. This means that the 7th-12th cpus (cpu6-cpu11) will not be assigned interrupts.

It sounds like you want to allow cpu0 and cpu1 to receive interrupts but prevent cpu2, cpu3, cpu4, and cpu5 from being assigned interrupts. That means you need two zeroes and four ones, or 111100. This is 3C in hexadecimal. So, you'd create /etc/default/irqbalance with the contents

ENABLED="1"
ONESHOT="0"
IRQBALANCE_BANNED_CPUS="3f"

To see what is going on, try

$ sudo service irqbalance stop
Stopping SMP IRQ Balancer: irqbalance.
$ source /etc/default/irqbalance 
$ sudo irqbalance --debug

Solution 2

an irqbalance bug prevents IRQBALANCE_BANNED_CPUS from working on NUMA machinges: http://code.google.com/p/irqbalance/issues/detail?id=43

As of now, 1.0.5 is the latest irqbalance release and does not have the fix.

Comments

[TraderJoeChicago]

Thanks for the explanation. The confusion here is: To BAN CPU 0 on a 6-cpu machine. Do i use: 000001 (1) or 111110 (3e). It is the first, right?

[TraderJoeChicago]

Please check my edit in the original question. I need to know how to read /proc/interrupts to make sure my configuration is working and my cpu is banned from interrupts. Thanks!

[TraderJoeChicago]

See my edit number 3: IRQBALANCE_BANNED_CPUS is being ingored on Ubuntu, unless we are messing up the mask. But I tried 1 and 3e without any success. :(

[sciurus]

See my debugging edits.

[TraderJoeChicago]

I did NOT use the quotes. Could that be it? :) I will test later and let you know...

[TraderJoeChicago]

NOPE. I can confirm it does NOT work on Ubuntu 10.04.1. This config option IRQBALANCE_BANNED_CPUS="1" is ignored.