Re: xhci_hcd intterrupt affinity in Dom0/DomU limited to single interrupt

[Boris Ostrovsky <boris.ostrovsky@oracle.com>]

On 09/01/2015 04:56 PM, Konrad Rzeszutek Wilk wrote:
> On Tue, Sep 01, 2015 at 05:39:46PM +0000, Justin Acker wrote:
>> Taking this to the dev list from users.
>>
>> Is there a way to force or enable pirq delivery to a set of cpus as opposed to single device from being a assigned a single pirq so that its interrupt can be distributed across multiple cpus? I believe the device drivers do support multiple queues when run natively without the Dom0 loaded. The device in question is the xhci_hcd driver for which I/O transfers seem to be slowed when the Dom0 is loaded. The behavior seems to pass through to the DomU if pass through is enabled. I found some similar threads, but most relate to Ethernet controllers. I tried some of the x2apic and x2apic_phys dom0 kernel arguments, but none distributed the pirqs. Based on the reading relating to IRQs for Xen, I think pinning the pirqs to cpu0 is done to avoid an interrupt storm. I tried IRQ balance and when co
 nfigured/adjusted it will balance individual pirqs, but not multiple interrupts.
> Yes. You can do it with smp affinity:
>
> https://cs.uwaterloo.ca/~brecht/servers/apic/SMP-affinity.txt
>
> I don't follow the "behavior seems to pass through to the DomU if pass through is enabled" ?
>
>>
>>
>> With irqbalance enabled in Dom0:
> What version? There was a bug in it where it would never distribute the IRQs properly
> across the CPUs.
>
> Boris (CC-ed) might remember the upstream patch that made this work properly?

I think we ended up taking the latest version of irqbalance as the one 
that added support for Xen guests was still not quite working. Besides, 
that patch was for xen-dyn-events, not for pirqs.

-boris


>>             CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7
>>   76:      11304          0     149579          0          0          0          0          0  xen-pirq-msi       0000:00:1f.2
>>   77:       1243          0          0      35447          0          0          0          0  xen-pirq-msi       radeon
>>   78:      82521          0          0          0          0          0          0          0  xen-pirq-msi       xhci_hcd
>>   79:         23          0          0          0          0          0          0          0  xen-pirq-msi       mei_me
>>   80:         11          0          0          0          0        741          0          0  xen-pirq-msi       em1
>>   81:        350          0          0          0       1671          0          0          0  xen-pirq-msi       iwlwifi
>>   82:        275          0          0          0          0          0          0          0  xen-pirq-msi       snd_hda_intel
>>
>> With native 3.19 kernel:
>>
>> Without Dom0 for the same system from the first message:
>>
>> # cat /proc/interrupts
>>             CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7
>>    0:         33          0          0          0          0          0          0          0  IR-IO-APIC-edge      timer
>>    8:          0          0          0          0          0          0          1          0  IR-IO-APIC-edge      rtc0
>>    9:         20          0          0          0          0          1          1          1  IR-IO-APIC-fasteoi   acpi
>>   16:         15          0          8          1          4          1          1          1  IR-IO-APIC  16-fasteoi   ehci_hcd:usb3
>>   18:     703940       5678    1426226       1303    3938243     111477     757871        510  IR-IO-APIC  18-fasteoi   ath9k
>>   23:         11          2          3          0          0         17          2          0  IR-IO-APIC  23-fasteoi   ehci_hcd:usb4
>>   24:          0          0          0          0          0          0          0          0  DMAR_MSI-edge      dmar0
>>   25:          0          0          0          0          0          0          0          0  DMAR_MSI-edge      dmar1
>>   26:      20419       1609      26822        567      62281       5426      14928        395  IR-PCI-MSI-edge      0000:00:1f.2
>>   27:   17977230     628258   44247270     120391 1597809883   14440991  152189328      73322  IR-PCI-MSI-edge      xhci_hcd
>>   28:        563          0          0          0          1          0          6          0  IR-PCI-MSI-edge      i915
>>   29:         14          0          0          4          2          4          0          0  IR-PCI-MSI-edge      mei_me
>>   30:      39514       1744      60339        157     129956      19702      72140         83  IR-PCI-MSI-edge      eth0
>>   31:          3          0          0          1         54          0          0          2  IR-PCI-MSI-edge      snd_hda_intel
>>   32:      28145        284      53316         63     139165       4410      25760         27  IR-PCI-MSI-edge      eth1-rx-0
>>   33:       1032         43       2392          5       1797        265       1507         20  IR-PCI-MSI-edge      eth1-tx-0
>>   34:          0          1          0          0          0          1          2          0  IR-PCI-MSI-edge      eth1
>>   35:          5          0          0         12        148          6          2          1  IR-PCI-MSI-edge      snd_hda_intel
>>
>>
>> The USB controller is an Intel C210:
>>
>> 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI])
>>      Subsystem: Dell Device 053e
>>      Flags: bus master, medium devsel, latency 0, IRQ 78
>>      Memory at f7f20000 (64-bit, non-prefetchable) [size=64K]
>>      Capabilities: [70] Power Management version 2
>>      Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
>>      Kernel driver in use: xhci_hcd
>>      Kernel modules: xhci_pci
>>       On Tuesday, September 1, 2015 11:50 AM, Ian Campbell <ian.campbell@citrix.com> wrote:
>>     
>>
>>   On Tue, 2015-09-01 at 13:56 +0000, Justin Acker wrote:
>>> Thanks Ian,
>>>
>>> I appreciate the explanation. I believe the device drivers do support
>>> multiple queues when run natively without the Dom0 loaded. The device in
>>> question is the xhci_hcd driver for which I/O transfers seem to be slowed
>>> when the Dom0 is loaded. The behavior seems to pass through to the DomU
>>> if pass through is enabled. I found some similar threads, but most relate
>>> to Ethernet controllers. I tried some of the x2apic and x2apic_phys dom0
>>> kernel arguments, but none distributed the pirqs. Based on the reading
>>> relating to IRQs for Xen, I think pinning the pirqs to cpu0 is done to
>>> avoid an I/O storm. I tried IRQ balance and when configured/adjusted it
>>> will balance individual pirqs, but not multiple interrupts.
>>>
>>> Is there a way to force or enable pirq delivery to a set of cpus as you
>>> mentioned above or omit a single device from being a assigned a PIRQ so
>>> that its interrupt can be distributed across all cpus?
>> A PIRQ is the way an interrupt is exposed to a PV guest, without it there
>> would be no interrupt at all.
>>
>> I'm afraid I'm out of my depth WRT how x86/MSIs and Xen x86/PV pirqs
>> interact, in particular WRT configuring which set of CPUs can have the IRQ
>> delivered.
>>
>> If no one else chimes in soon I'd suggest taking this to the dev list, at
>> the very least someone who knows what they are talking about (i.e. other
>> than me) might be able to help.
>>
>> Ian.
>>
>>
>>
>>    
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel