* Collectl now support monitoring Interrupts by CPU
[not found] <477FDAA8.2030001@hp.com>
@ 2008-02-29 13:18 ` Mark Seger
0 siblings, 0 replies; only message in thread
From: Mark Seger @ 2008-02-29 13:18 UTC (permalink / raw)
To: util-linux-ng, linux-kernel, netdev
In keeping with the spirit of adding new useful features when they help
solve real world problems, I've added this ability to collectl as a
direct result of a problem we were recently having when doing some
network performance testing. It turned out all the interrupts were
being processed by cpu0 (this was on an 8-core system) but all collectl
told us was the aggregate number! Once we moved to a NIC/driver that
supported multiple queues that could distribute interrupts to multiple
CPUs it only made sense to enhance collectl to let us verify that this
was indeed happening - I personally find examining /proc/interrupts for
changes more trouble than it's worth.
In any event, the following is an example of how collectl can present
this data, first summarized by CPU:
#Time Cpu0 Cpu1 Cpu2 Cpu3 Cpu4 Cpu5 Cpu6 Cpu7
12:49:55 4828 16K 1000 16K 18 16K 16K 0
12:49:56 4804 16K 1000 16K 0 16K 16K 0
12:49:57 4811 16K 1000 16K 18 16K 16K 0
12:49:58 4789 16K 1000 16K 0 17K 16K 44
and here by the type of the interrupt itself:
# INTERRUPT DETAILS
# Int Cpu0 Cpu1 Cpu2 Cpu3 Cpu4 Cpu5 Cpu6 Cpu7
Type Device(s)
12:48:50 082 0 0 0 7731 0 0 0
0 PCI-MSI-X eth2 (queue 0)
12:48:50 098 0 0 0 0 2037 0 0
0 PCI-MSI-X eth2 (queue 2)
12:48:50 122 0 0 2240 0 0 0 0
0 PCI-MSI-X eth2 (queue 5)
12:48:50 138 0 7084 0 0 0 0 0
0 PCI-MSI-X eth2 (queue 7)
12:48:50 154 0 0 0 0 0 7723 0
0 PCI-MSI-X eth3 (queue 0)
12:48:50 162 9082 0 0 0 0 0 0
0 PCI-MSI-X eth3 (queue 1)
12:48:50 178 0 0 0 0 0 0 8253
0 PCI-MSI-X eth3 (queue 3)
12:48:50 210 0 0 0 0 0 0 0
6417 PCI-MSI-X eth3 (queue 7)
12:48:50 218 1 0 0 0 0 0 0
0 PCI-MSI eth0
You can also look at all CPU loads and interrupts together like this:
# SINGLE CPU STATISTICS
# CPU USER NICE SYS WAIT IRQ SOFT STEAL IDLE INTRPT
07:09:28 0 0 0 0 0 0 0 0 100 16
07:09:28 1 0 0 0 0 0 0 0 100 0
07:09:28 2 0 0 0 0 0 0 0 100 999
07:09:28 3 0 0 0 0 0 0 0 100 16
For those not familiar with collectl, you can collect virtually
everything all the existing linux 'stat' utilitie provide plus a lot
more such as processes (including I/O stats), Infininband, Quadrics,
Slab, Lustre and more. Plus a lot more feature too numerous to list but
there's a pretty good summary here -
http://collectl.sourceforge.net/Features.html
But don't take my word for it, check out the website at
http://collectl.sourceforge.net/ where you can see a pretty good set of
examples in the documentation and even follow the tutorial.
-mark
^ permalink raw reply [flat|nested] only message in thread