Is there a utility that interprets /proc/interrupts data over time?

Question

Is there something out there? Like top is for ps

Solution 1

watch -n0.1 --no-title cat /proc/interrupts

Solution 2

dstat can also be used for that.

dstat -tif 60

To list all the interrupts (those with more than 10 in /proc/stat)

dstat -tf --int24 60

Same but using /proc/interrupts, so include things like LOC, PMI, RES...

You can also select the list of those you want:

$ dstat -t --int24 -I23,LOC,RES 5
----system---- ----interrupts---
     time     |  23   LOC   RES
21-12 16:30:23|   2   489    52
21-12 16:30:28|  30   593     6
21-12 16:30:30|  37   929     7

See also --top-int to track the most active interrupt:

$ dstat -t --top-int
----system---- ---most-frequent----
     time     |     interrupt
21-12 16:33:21|5242880-edge enp10s0  56
21-12 16:33:22|5242880-edge enp10s0  68
21-12 16:33:23|5242880-edge enp10s0   4
21-12 16:33:24|5242880-edge enp10s0   3
21-12 16:33:25|5242880-edge enp10s0  61
21-12 16:33:26|5242880-edge enp10s0  11
21-12 16:33:27|512000-edge ahci[0000:00:1f.2]   5
21-12 16:33:28|5242880-edge enp10s0  52
21-12 16:33:29|5242880-edge enp10s0  20
21-12 16:33:30|32768-edge i915  57

Solution 3

For a server with lot's of CPUs, I found https://github.com/lanceshelton/irqstat very useful. It shows where interrupts are happening in real time:

Sun Oct 21 20:16:09 2018
IRQs / 5 second(s)
IRQ#  TOTAL  NODE0   NODE1  NAME
  35  38060  38060       0  IR-PCI-MSI 2621440-edge enp5s0-rx-0
  36  19853  19853       0  IR-PCI-MSI 2621441-edge enp5s0-tx-0
  34    311    311       0  IR-PCI-MSI 512000-edge ahci[0000:00:1f.2]
  29    105    105       0  IR-PCI-MSI 2097152-edge enp4s0-rx-0
  42      0      0       0  IR-PCI-MSI 77824-edge ioat-msix
  43      0      0       0  IR-PCI-MSI 79872-edge ioat-msix
  49      0      0       0  IR-PCI-MSI 67180544-edge ioat-msix
  52      0      0       0  IR-PCI-MSI 67186688-edge ioat-msix
  53      0      0       0  IR-PCI-MSI 67188736-edge ioat-msix
  48      0      0       0  IR-PCI-MSI 67178496-edge ioat-msix

Solution 4

mpstat(1) N M -I lets you do this with a specified polling interval and number of reports.

• N is the polling interval, in seconds.
• M is the number of times to report.
• According to the man page, -I, which takes a number of options, is to "Report interrupts statistics".

Furthermore,

intr/s
Show the total number of interrupts received per second by the CPU or CPUs.
With the CPU keyword, the number of each individual interrupt received per second by the CPU or CPUs is displayed. Interrupts are those listed in /proc/interrupts file.

View more solutions

Comments

[Steven D]

Usually, it is better to add your answer as an actual answer rather than editing the question.

[Admin]

This is very hard to use if you have many threads, it wraps the screen.

[Admin]

Could not get this to work properly. There is a nasty unstable bug with ruby-curses that shoots a bunch of warnings all over the screen. bugs.debian.org/cgi-bin/bugreport.cgi?bug=958973 I am going to admit I could try to spend time fixing this but that involves setting up ruby / gem / other supported packages and env. Rather find a solution that is native to most linux systems.

[Admin]

@Dave You are trying to use the wrong irqtop! For comparison, on a Fedora 35 system: rpm -qf $(which irqtop) -> util-linux-2.37.4-1.fc35.x86_64 and file $(which irqtop) -> /usr/bin/irqtop: ELF 64-bit LSB pie executable [..] IOW, try the irqtopq that is part of the util-linux package!