Re: [PATCH RFC 1/1] genirq: Make threaded handler use irq affinity for managed interrupt

John Garry
2019-12-13T20:36:43Z

Hi Marc,

>> JFYI, we're still testing this and the patch itself seems to work as
>> intended.
>>
>> Here's the kernel log if you just want to see how the interrupts are
>> getting assigned:
>> https://pastebin.com/hh3r810g
>
> It is a bit hard to make sense of this dump, specially on such a wide
> machine (I want one!)

So do I :) That's the newer "D06CS" board.

without really knowing the topology of the system.

So it's 2x socket, each socket has 2x CPU dies, and each die has 6
clusters of 4 CPUs, which gives 96 in total.

>
>> For me, I did get a performance boost for NVMe testing, but my
>> colleague Xiang Chen saw a drop for our storage test of interest  -
>> that's the HiSi SAS controller. We're trying to make sense of it now.
>
> One of the difference is that with this patch, the initial affinity
> is picked inside the NUMA node that matches the ITS.

Is that even for managed interrupts? We're testing the storage
controller which uses managed interrupts. I should have made that clearer.

In your case,
> that's either node 0 or 2. But it is unclear whether which CPUs these
> map to.
>
> Given that I see interrupts mapped to CPUs 0-23 on one side, and 48-71
> on the other, it looks like half of your machine gets starved,

Seems that way.

So this is a mystery to me:

[ 23.584192] picked CPU62 IRQ147

147: 0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 ITS-MSI 94404626
Edge hisi_sas_v3_hw cq

and

[ 25.896728] picked CPU62 IRQ183

183: 0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 ITS-MSI 94437398
Edge hisi_sas_v3_hw cq

But mpstat reports for CPU62:

12:44:58 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %gnice %idle
12:45:00 AM 62 6.54 0.00 42.99 0.00 6.54 12.15 0.00
0.00 6.54 25.23

I don't know what interrupts they are...

It's the "hisi_sas_v3_hw cq" interrupts which we're interested in.

and that
> may be because no ITS targets the NUMA nodes they are part of.

So both storage controllers (which we're interested in for this test)
are on socket #0, node #0.

It would
> be interesting to see what happens if you manually set the affinity
> of the interrupts outside of the NUMA node.
>

Again, managed, so I don't think it's possible.

Thanks,
John